break apart illinois-common-resources

[cogcomp-dev]

...into several smaller pieces. Gazetteers can go in one; brown clusters (and maybe other clusters) in another. This may help reduce the problems with large dependencies in CI.

[cowchipkid]

I would suggest anything approaching 100M within this jar file warrants it's own jar file.

[mssammon]

There's already a version just for SRL; please check and see if this overlaps with the NER when deciding how to break down the common-resources jar. The SRL resources jar itself may merit further segmentation. Before this ticket is closed (after testing NER -- including running the training/evaluation) please open a ticket on the illinois-srl project to migrate to the new jars, if the corresponding resources have been affected.

[shatu]

NER uses an older version of illinois-common-resources (1.3). So, which one should I break apart -- the one that NER uses, or the latest one?

Also, what should be the right approach :-

1. Should we keep the current illinois-common-resources as it is and create new jars containing portions of it? Or,
2. Should we modify the current illinois-common-resources to contain only a portion of its current members and move the rest to separate jars. -- This will probably work only if nothing else depends on common-resources. Or, we'll have to open issue in all such projects to update their dependencies correspondingly.

[shatu]

BTW, I can also see a illinois-common-resources-1.5-ner jar in the repo .. which is around 200M .. any idea what is that being used for?

[mssammon]

I think that was me trying to generate an ner-only jar.

[shatu]

Okay, so can someone familiar with the following resources, suggest me a logical partitioning, along with the suggested names of the new jars :-

Size -- Name (File/Dir)
64K -- levin-verbClass.txt (File)
504K -- verb-lemDict.txt (File)
12M -- cbcData (Dir)

145M -- CORLEX (Dir)

91M -- WordEmbedding (Dir)

2.6M -- rogetThesaurus (Dir)
97M -- gazetteers (Dir)

144M -- lin-clusters (Dir)
57M -- brown-clusters (Dir)

217M -- lin-similarity (Dir)

Also, kindly address the queries posed in the previous comment.

Thanks,
Shashank

[cowchipkid]

Definately used the 1.5 version, it includes an updated gazetteer.

I can only speak to a few of resources, but these are the ones I know are used by NER, although there may be others. The brown-clusters and gazetteers are well named and these are used in NER. I would suggest for these two datums, you split them each into their own jar files, I suspect there are cases where each are used independently. I would name them gazetteers.jar and brown-clusters.jar.

I would suggest you retain the structure within the jar to minimize code changes. In that way, all we should have to change is the pom.xml files…

[danyaljj]

Briefly repeating Shashank, here is what the file contains:

My suggestion is one jar file per resource. For example I would put brownCluster folder inside a jar artifact with the same name (and retain the current version 1.3).

[danyaljj]

Something to notice though: the latest version number on m2repo is 1.5, not 1.3 (which is currently being used inside the Edison code).
Also there three files:

• illinois-common-resources-1.5-illinoisSRL.jar
• illinois-common-resources-1.5-ner.jar
• illinois-common-resources-1.5.jar

which seem to have some commonality. Here is what I see inside each: