Lexical Tools - build Processes

This pages describes the build, development and test procedures on Lexical Tools annual releases:

• development local build (ant) & test
• production Ci-Cd build (mvn) & test
• Ci-Cd preprocess (Makefile)
• Ci-Cd pipleline (git)

• I. Development local build (Ant) & test
  • Ant Build:

    Command	setup	Operation results
    shell> ant clean	build.xml	remove ./classes
    shell> ant	build.xml	create ./classes create ./lib/lvg${YEAR}api.jar
    shell> ant dist	build.xml	create ./classes create ./lib/lvg${YEAR}api.jar create ./lib/lvg${YEAR}dist.jar
    shell> ant release	build.xml	create ./classes create ./lib/lvg${YEAR}api.jar create ./lib/lvg${YEAR}dist.jar create ./lib/lvg${YEAR}lite.tgz
    shell> packLvgFull ${YEAR}	use ./lib/lvg${YEAR}dist.jar	create ./lvg${year}.tgz
    shell> packLvgLite ${YEAR}	use ./lib/lvg${YEAR}dist.jar	create ./lvg${year}lite.tgz

  • Test:

    Test Type	setup	Operation results
    unit test	install ./lvg${YEAR}.tgz to ${PROJECT}	check results on all flows and options
    lite test	unzip ./lib/lvg${YEAR}Lite.tgz	Check all scripts

• II. Production Ci-Cd build (mvn) & test
  • pom.xml:
    • version:
      • use ${YEAR}.0-SNAPSHOT for snapshot build
        => for snapshot development build, multiple versions is ok
      • use ${YEAR}.0 for official release build
        only 1 version is allowed in Nexus.
        => delete artifact (*.jar) of ${YEAR}.0 in Nexus -> lhc-lexicon-maven-release -> gov -> nih -> nlm -> lvg -> ${YEAR}.0 (before use mvn for release build)
  • Maven & pom.xml updates:
    Use the latest version on Maven Central Repository

    groupId	artifactId	version	Notes
    gov.nih.nlm	lvg	${YEAR}.0-SNAPSHOT ${YEAR}.0	SNAPSHOT is used for development.
    com.ibm.icu	icu4j	73.2	Unicode package
    org.hsqldb	hsqldb	2.7.2	HsqlDb database (use jdk8)
    junit	junit	4.13.2	for unit tests
    org.apache.maven.plugins	maven-compiler-plugin	3.11.0	maven compiler
    org.apache.maven.plugins	maven-jar-plugin	3.3.0	Maven manifest
    Profile
    org.apache.maven.plugins	maven-assembly-plugin	3.6.0	mvn package -Psingle generates ./target/lvg-${YEAR}.0-jar-with-depednencies.jar
    org.apache.maven.plugins	maven-resources-plugin	3.3.1	copy resource files
    org.apache.maven.plugins	maven-shade-plugin	3.5.1	pack all dependecies into one artifact (uber-jar). mvn package -Pshade generates ./target/lvg-${YEAR}.0-shade.jar mvn deploy -Pshade-attached attached (not replace) the original artifact need to exclude META-INF/MANIFEST.MF from all artifacts to avoid duplciates warning
    org.apache.maven.plugins	maven-javadoc-plugin	3.6.0	mvn package -P javadoc generate ./target/apidocs/* (not used in the release)
    org.apache.maven.plugins	maven-source-plugin	3.3.0	mvn package -P source generates lvg-{YEAR}.0-sources.jar (include source)

  • Maven basic cycle:

    Command	setup	Operation results
    shell> mvn clean	pom.xml	remove ./target
    shell> mvn validate	pom.xml	validate maven setup
    shell> mvn compile	pom.xml	create ./target
    shell> mvn test	pom.xml ./test/java/*.java	test junit
    shell> mvn package	pom.xml	create ./target/lvg-${YEAR}.0.SNAPSHOT.jar or create ./target/lvg-${YEAR}.0.jar
    shell> mvn install	pom.xml	publish artifacts to maven local repository ./lib/*.jar to ~/.m2/repository/..
    shell> mvn deploy	pom.xml	publish artifacts to maven remote repository (LHC) ./lib/*.jar to https://lhc-nexus.nlm.nih.gov/repository/..

  • Maven Ci-Cd commands:

    Command	setup	Operation results
    shell> mvn deploy -Pshade	pom.xml	create lvg-${YEAR}.0-SNAPSHOT-shaded.jar publish artifacts to maven remote repository (LHC) ./lib/*.jar to https://lhc-nexus.nlm.nih.gov/repository/..
    shell> packLvgFull ${YEAR} TRUE	use ./target/lvg-${YEAR}.0-shaded.jar to replace ./lib/lvg${YEAR}dist.jar	create ./lvg${year}.tgz
    shell> packLvgLite ${YEAR} TRUE	use ./target/lvg-${YEAR}.0-shaded.jar to replace ./lib/lvg${YEAR}dist.jar	create ./lvg${year}lite.tgz

  • Test:

    Test Type	setup	Operation results
    unit test	install lvg to ${PROJECT}	check results on all flows and options
    lite test	unzip lvgLite	Check all scripts

• III. Ci-Cd preprocess - Makefile
  • .gitlab-ci.yml
    => defines CI-CD pipeline process - build, packLite, packFull
  • Makefile
    => a file called by ./gitlab-ci to define tasks on all phases in the CI-CD pipeline

    variable	operations
    login	login to docker
    build	use maven to build and deploy lvg artifacts to LHC-Nexus
    packLite	pack lite (min.) Lvg to ./lvgLite${YEAR}.tgz
    packFull	pack full Lvg to ./lvg${YEAR}.tgz

  • packLite & packFull:
    • Variable defintion:

      Variable name	function	Notes
      ${VERSION}	lvg version	updated in the Makefile
      ${LVG_PACK}	pack lvg${YEAR}lite.tgz and lvg${YEAR}.tgz	TRUE: default
      ${LVG_UPLOAD}	upload to Nexus	FALSE: development (default) TRUE: release Used for Website download and WebLvg if the *.tgz files exist in the Nexus, the upload will be failed. delete artifact (*.tgz) of ${YEAR} in Nexus -> lhc-lexicon-raw -> www -> lvg -> ${YEAR} (before use mvn for release build) delete artifacts of the *.jar (release files) for deploying the same release version. for snapshot development build is ok
      ${LVG_MVN_SNAPSHOT}	snapshot build	TRUE: development (default) FALSE: release
      ${LVG_MVN_JAR}	Use mvn to build	TRUE: maven build FALSE: ant build (default)

    • The above variables are used in Git and CiCd. They are deinfed in the environment variables (for linux) or in Git-> Lexicon -> Setting -> CI/CD -> Variables -> Reveal Values
    • Quick variable setup table:

      Variable name	Dev - Ant	Maven/PreProcess	Production
      ${YEAR}	year	year	year
      ${LVG_PACK}	TRUE	TRUE	TRUE
      ${LVG_UPLOAD}	FALSE	FALSE	TRUE
      ${LVG_MVN_SNAPSHOT}	TRUE	TRUE	FALSE
      ${LVG_MVN_JAR}	FALSE	TRUE	TRUE

  • IV. Ci-Cd pipleline - Git
    • Development & Test - Git
      • develope and test on version of ${YEAR}.0-SNAPSHOT in pom.xml
      • after complete above, update version to ${YEAR}.0 in pom.xml for final deploy version)
      • Git commands:

        command	desciption	notes
        git branch	check which branch is on
        git checkout develop	Use "develop" for development	development uses snapshot build
        git add -A git commit -m "LEX_xxx, msg git push origin	save software changes to Git remote
        git tag	show all tags	tag is used to start the Ci-Cd build
        git tag -a v.${YEAR}.0 -m "msg"	tag the current version
        git push origin tagName	push a tag to remote origin	A new tag will start the Ci-Cd pipleline
        git push origin -- tags	pushshow all tags to remote origin

      • Tests:
        => use script downloadFiles ${YEAR} TRUE to download *.tgz from LHC-nexus to ./download
        • lvg${YEAR}.tgz => unit test, platform test, performance test, etc.
        • lvgLite${YEAR}.tgz => lite test

        Test Type	setup	Operation results
        unit test	install lvg to ${PROJECT}	check results on all flows and options
        lite test	unzip lvgLite	Check all scripts
        GUI Lexical Tool test	unzip lvgLite	Check all scripts
        Perfromance test	unzip lvgLite	Check all scripts
        Other tests	unzip lvgLite	Check all scripts

    • Production release
      • For the release, need to remove the following assets (if they exists) in the Nexus before a new (final) Ci-Cd build
        • LVG package (annual release)
          ${NEXUS_URL}/repository/lhc-lexicon-maven-releases/gov/nih/nlm/lvg/${YEAR}.0/
          • lvg-${YEAR}.0.jar
          • ...
        • LVG artifacts (used by WebLvg)
          ${NEXUS_URL}/repository/lhc-lexicon-raw/www/lvg/${YEAR}/
          • lvg${YEAR}.tgz
          • lvg${YEAR}lite.tgz
      • Git commands:
        • update the pom.xml for mvn build to spanshot sand ${LVG_MVN_SANPSHOT)=TRUE if release is already build in develop.
        • update ${LVG_UPDLOAD}=FALSE if release is already uploaded to Nexus in develop.

        Git command	Descriptions	Notes
        git checkout master	switch to master
        git merge develop	merge changes in dev to master
        git tag	show all tags
        git tag -a v.${YEAR}.01 -m "msg"	tag the current version	tag is used to save the official release (not to start a Ci-Cd pipeline).

      • Tests: unit tests, lite test, GUI Lexical Tool test, Platform test.