The Holy Java

Building the right thing, building it right, fast

Posts Tagged ‘deployment’

The Risks Of Big-Bang Deployments And Techniques For Step-wise Deployment

Posted by Jakub Holý on February 17, 2014

If you ever need to persuade management why it might be better to deploy a larger change in multiple stages and push it to customers gradually, read on.

A deployment of many changes is risky. We want therefore to deploy them in a way which minimizes the risk of harm to our customers and our companies. The deployment can be done either in an all-at-once (also known as big-bang) way or a gradual way. We will argue here for the more gradual (“stepwise”) approach.

Big-bang or stepwise deployment?

A big-bang deployment seems to be the natural thing to do: the full solution is developed and tested and then replaces the current system at once. However, it has two crucial flaws.

Read the rest of this entry »

Posted in SW development | Tagged: , | Comments Off

Using Java as Native Linux Apps – Calling C, Daemonization, Packaging, CLI (Brian McCallister)

Posted by Jakub Holý on September 25, 2012

This is a summary of the excellent JavaZone 2012 talk Going Native (vimeo) by Brian McCallister. Content: Using native libraries in Java and packaging them with Java apps, daemonization, trully executable JARs, powerful CLI, creating manpages, packaging natively as deb/rpm.

1. Using Native Libs in Java

Calling Native Libs

Calling native libraries such as C ones was hard and ugly with JNI but is very simple and nice with JNA (GPL) and JNR (Apache/LGPL)
Read the rest of this entry »

Posted in Languages | Tagged: , , , , , | Comments Off

Releasing a project to Maven Central repository via Sonatype

Posted by Jakub Holý on February 7, 2010

If you have an open-source project and want it published into the Maven Central repository – even if it itself isn’t build with maven – to make it visible to all Maven users without any special effort (at least once you set everything up), read on.

This is excatly what I wanted to do with my dbunit-embeddedderby-parenttest, an extension of DbUnit that extremly simplifies setup of unit tests that need a database by using an embedded Derby DB and providing suitable defaults and utility methods.

The essential step is to deploy your artifacts to the Sonatype’s OSS hosting repository, which is free for open-source project, and which will take care of hourly automatic synchronization with Maven Central once you release your artifacts to Sonatype, satisfy some requirements and let them know about it. There is a good guide for setting up hosting with Sonatype but there a few tricky parts and that’s why I’ve written down this blog.

The procedure is as follows:

  1. Mavenize your project. You don’t actually need to build it with Maven, you only need to provide a maven metadata file (pom.xml). You may want to check my project’s pom.xml v1.0.2 (or the latest) for inspiration and to see how I’ve configured the other things mentioned here.
  2. Follow the steps described in the guide for setting up hosting with Sonatype:
    1. Sign up on the Sonatype repository manager Nexus at http://oss.sonatype.org/
    2. Sign up on the Sonatype ticketing system at https://issues.sonatype.org/browse/OSSRH
    3. Create a new ticket asking the creation of a maven repository for your project, see my OSSRH-141 as an example
    4. Make sure that your pom.xml satisfies all the requirements for publishing to the Maven Central Repo, listed in the guide (SCM url, site url, no <repositories> etc.)
    5. Wait for creation of the repositories, once finished, a comment will be added to the ticket, which should also result in send an email notification to you
  3. Build and sign your artifacts (see below)
  4. Deploy your artifacts to the new repository at Sonatype (see below)
  5. (optional?) Promote the published artifacts from the staging to the release repository (only applicable if you do not deploy directly to the release repo – but it seems obligatory to use staging now.)
  6. When the artifacts are in the release repository, add a comment to the original ticket asking for them to be published to Maven Central. For the first time they need to do it manually but since that on it will be synchronized automatically every hour

Build and sign your artifacts

Build your project’s artifacts as you are used to, for example with mvn package if you’re a Maven guy or with ant.

Next you need to sign them with your private key because valid signatures are one of the requirements for publishing to Maven Central. This is already well described in the article How to Generate PGP Signatures with Maven so I’ll only summarize it and add few tips.

  1. Create a new GPG or PGP key if you haven’t one already. You can use the command-line application gpg or a GUI tool such as seahorse (under Ubuntu: Applications > Accessories > Passwords and Encryption Keys).
  2. Publish the key to the required PGP server pgp.mit.edu:
    gpg –keyserver pgp.mit.edu –send-key <ID of your key; for me it was 66AE163A>
  3. Sign your files using either gpg directly or the maven-gpg-plugin (also described in the article mentioned above) – see below.

Signing with maven-gpg-plugin and automatic deployment

Using the maven-gpg-plugin has the advantage that running

mvn deploy -DperformRelease=true -Dgpg.passphrase=**** -Dgpg.keyname=<your key ID>

will automatically sign your artifacts and deploy them together with the signatures. On the other hand, as with any plugin, its configuration and use may have some issues. (Setting the property performRelease=true is necessary in my case because I have configured the plugin to be actived only if this is set.)

Tips

  • If the key you want to use for signing isn’t your defaul key then you need to tell Maven which key to use by storing its id in the property gpg.keyname either by passing -Dgpg.keyname=… on the command line or setting it in the pom.xml as you can is in the mine.

Signing and deploying signatures manually

You may instead decide to sign your artifacts manually using directly gpg (or whatever tool you have), for example like this:

gpg -u <your key ID> –sign –detach-sign -a target/dbunit-embeddedderby-parenttest.jar

This command would create the signature file target/dbunit-embeddedderby-parenttest.jar.asc.

Next you will need to deploy the signature to the repository along with the original artifact:

mvn deploy:deploy-file -Durl=http://oss.sonatype.org/service/local/staging/deploy/maven2 -DrepositoryId=nexus-releases -Dfile=target/dbunit-embeddedderby-parenttest.jar.asc -DpomFile=pom.xml -Dpackaging=asc

  • The url will be the same for everybody who wants to publish artifacts via a staging repository and not directly to to the release one otherwise you need to provide the URL of the release repository created for you by Sonatype.
  • The repositoryId must correspond to a <server> record in your <user home>/.m2/.settings.xml with the same id and your login name and password  for oss.sonatype.org, it tells maven what credentials to use to authenticate with the target repository
  • Beware that the name of the uploaded file isn’t the same as of the local file being uploaded but is constructed by Maven from the artifactId, version, and optionally packaging and classifier provided in the pom.xml or on the command line

And perhaps also deploy the artifact itself if not done otherwise:

mvn deploy:deploy-file
-Durl=http://oss.sonatype.org/service/local/staging/deploy/maven2
-DrepositoryId=nexus-releases
-Dfile=target/dbunit-embeddedderby-parenttest.jar -DpomFile=pom.xml

Examples of manual deployment

The following examples show you how to deploy files if different names to the Sonatype’s staging repository. They all use a common pom.xml (you’ll perhaps want change the groupId and artifactId) and you will need to change the -DrepositoryId to correspond to the server id you’ve assigned to your Sonatype repository credentials in your settings.xml (discussed below).

The common pom.xml:

 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
        <modelVersion>4.0.0</modelVersion>
        <groupId>eu.ibacz.pokusy.jakubholy</groupId>
        <artifactId>smazMe</artifactId>
        <name>dummy pokusny projekt</name>
        <version>1.0-SNAPSHOT</version>       
</project>
$ mvn deploy:deploy-file -Durl=http://oss.sonatype.org/service/local/staging/deploy/maven2 -DrepositoryId=nexus-releases -Dfile=smazMe.jar -DpomFile=pom.xml -Dpackaging=jar

=> Uploading: http://oss.sonatype.org/service/local/staging/deploy/maven2/eu/ibacz/pokusy/jakubholy/smazMe/1.0-SNAPSHOT/smazMe-1.0-20100210.143500-1.jar

$ mvn deploy:deploy-file-Durl=http://oss.sonatype.org/service/local/staging/deploy/maven2 -DrepositoryId=nexus-releases -Dfile=smazMe.jar.asc -DpomFile=pom.xml

=> Uploading: http://oss.sonatype.org/service/local/staging/deploy/maven2/eu/ibacz/pokusy/jakubholy/smazMe/1.0-SNAPSHOT/smazMe-1.0-20100210.143508-2.jar.asc

 $  mvn deploy:deploy-file -Durl=http://oss.sonatype.org/service/local/staging/deploy/maven2 -DrepositoryId=nexus-releases -Dfile=smazMe-all.zip -DpomFile=pom.xml -Dpackaging=zip -Dclassifier=all

=> Uploading:

http://oss.sonatype.org/service/local/staging/deploy/maven2/eu/ibacz/pokusy/jakubholy/smazMe/1.0-SNAPSHOT/smazMe-1.0-20100210.143513-3-all.zip

$  mvn deploy:deploy-file -Durl=http://oss.sonatype.org/service/local/staging/deploy/maven2 -DrepositoryId=nexus-releases -Dfile=smazMe-all.zip.asc -DpomFile=pom.xml -Dpackaging=zip.asc -Dclassifier=all

=> Uploading:

http://oss.sonatype.org/service/local/staging/deploy/maven2/eu/ibacz/pokusy/jakubholy/smazMe/1.0-SNAPSHOT/smazMe-1.0-20100210.143524-4-all.zip.asc

Note: You may also supply the option -DuniqueVersion=false to avoid the addition of a generated number to file names.

Deploy your artifacts

Since we are going to deploy our artifacts to a Maven repository, we need to use Maven for that. We have a couple of ways of doing that: we can let Maven to deploy all that is necessary (which will also build and sign the artifact, if necessary), alternatively we can deploy already prepared artifacts manually, one by one, or finally we can do any of these using Maven Ant integration.

Notice that it’s now required to publish artifacts via a staging repository instead
of directly to the release one so that you can review and have to approve them before they are actually release (details in the following section).

A. Deploying directly with Maven

Unless you decide to use the release plugin (which I do use), which does some neat and useful things such as creating an SVN tag, increasing the project’s version etc., the deployment is as easy as executing

mvn deploy -DperformRelease=true -Dgpg.passphrase=**** -Dgpg.keyname=<your key ID>

(see the section about maven-gpg-plugin to understand why I need these three properties).

There are two important configurations that make this possible. First of all, you must configure the target deployment repository in your pom.xml:

<project ...>
   ...
    <distributionManagement>
        <repository>
            <id>nexus-releases</id>
            <name>Nexus Release Repository</name>
            <url>http://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
        </repository>
    </distributionManagement>
</project>

The <id> is arbitrary and the <url> is same for all, it’s actually a kind of a virtual repository and the artifacts will be in reality deployed to a temporary repository based on your username.

Next you need to store your username and password for oss.sonatype.org under the same <id> as above in the file <user home>/.m2/settings.xml so that maven can authenticate with the repository:

<settings>
   ...
   <servers>
      <server>
         <id>nexus-releases</id>
         <username>malyvelky</username>
         <password>******</password>
      </server>
   </servers>
</settings>

Troubleshooting deployment failures

Error 401 (Unauthorized)

If the deployement fails with an error like this:

Error deploying artifact: Failed to transfer file: http://oss.sonatype.org/service/local/staging/deploy/maven2//net/jakubholy/testing/dbunit-embeddedderby-parenttest/1.0.2/dbunit-embeddedderby-parenttest-1.0.2.jar. Return code is: 401

(notice the error code 401) it most likely means that you haven’t configured your username/password correctly or that the two IDs (repository id in pom.xml and server id in settings.xml) do not match.

Error 400 (Bad Request)

This error indicates that the repository manager has rejected your artifact for some reason. This may happen for example if you try to deploy a snapshot version into the staging repository, which is not allowed by its policy. (For snapshots you should reference the snapshot repository created for you as distributionManagement/snapshotRepository in your pom.xml.)

B. Deploying manually

If you have already built and signed your artifacts, you can deploy them manually with the mvn deploy:deploy-file goal as in the example below:

$ mvn deploy:deploy-file-Durl=http://oss.sonatype.org/service/local/staging/deploy/maven2 -DrepositoryId=nexus-releases -Dfile=smazMe.jar.asc -DpomFile=pom.xml

See the section Signing and deploying signatures manually above for explanation and examples of deployment of different types of files (.jar, .zip, signatures).

In this case you do not need the distributionManagement section in your pom.xml (because you’re providing all the information on the command line) but you still need to provide your login and password for the target repository (option -DrepositoryId=…) in your settings.xml as described above.

Of course you can run the deployment commands manually or for instance from Ant.

C. Deploying with Ant nad the Maven Ant Tasks

Update 10/2010: Sonatype user guide now contains the section 7c. Stage Artifacts with Ant.

You may also use the Maven Ant Tasks to invoke Maven operations from an Ant build including a task for file deployment. It does basically the same as invoking mvn deploy:deploy-file from the command line but you can provide the repository credentials here without a need for settings.xml and you may also find this way of Maven invocation nicer than calling a command-line application.

Promote the published artifacts from the staging to the release repository

This step is only necessary if you deploy via the staging repository and not directly to the release repository. But according to a recent email by Juven Xu it seems to be now obligatory to deploy via staging:

Staging is one of the most outstanding features in Nexus Pro. We’ve
been provided you this service for a long time, but maybe you were not
aware of this. So I changed your release repository configuration and
you will no longer be able to deploy release artifacts directly into
the release repository. Instead, you should deploy your release
artifacts into the staging repository: http://oss.sonatype.org/service/local/staging/deploy/maven2/,
then log in from Nexus UI, do things like closing staging repository,
and promoting the staged artifacts into your release repository.

If you are not familiar with Staging, it’s well documented at http://www.sonatype.com/books/nexus-book/reference/staging.html.

New instructions (6/2011)

  1. Log in to oss.sonatype.com
  2. Click on “Staging Repositories” under Build Promotion
  3. Verify the content of the repository (in the bottom pane), check it, click Close, confirm
  4. Check the repo again, click “Release”
  5. You shall now see your artifacts in the release repository created for you (mine is http://oss.sonatype.org/content/repositories/jakubholy-releases/).
  6. In some hours (?) it should also appear in Maven Central

Old instructions

You must do the following to promote the deployed artifacts from the staging to the release repository:

  1. Log in to oss.sonatype.com
  2. In the right menu, click on “Staging” in the box called “Enterprise” (#1 in the screenshot) => you should see the “staging profile” named <your project name> (for me it was “net.jakubholy”; see #2)
  3. Click on your staging repository target (“net.jakubholy”, #2) – you should see the individual repositories (some generated name such as net.jakubholy-028) within that including their status, which is “open” right after a deployment (which means that files may be still deployed and redeployed to this temporary repository)
  4. Verify that the project artifacts are as expected (see below)
  5. “Close” and “Promote” the staging repository to promote its content to the release repository: right-click on its status and you should see a little “context menu” with options Close and Drop, select Close (#3). You’ll be asked for a description, which will be associated with that release. After some time you should also receive a confirmation email from the “Nexus Repository Manager” with the subject “Nexus: Staging Completed.”
  6. “Promote” the closed staging repository to the release one in a way similar to closing it, only now seeing the current status as “closed” and selecting “Promote” from the context menu (beware: there may be some delay before this becomes possible), select your release repository as the target (Jakubholy Releases for me).
  7. You shall now see your artifacts in the release repository created for you (mine is http://oss.sonatype.org/content/repositories/jakubholy-releases/).

Sonatype Repository Manager - closing a stagin repo.

Required artifacts

[Since 2010-04-19] It is no required that you include -javadoc.jar along with -sources.jar with your artifacts otherwise you will not be allowed to promote. You can easily create them either with Maven or manually.

Verifying project artifacts before promoting from the staging repository

Before you promote the project’s artifacts from the staging to the release repository you may want to check few things to be sure that the release is OK:

  • Are all artifacts included?
  • pom.xml:
    • Is the version all right?
    • Are the SCM URLs correct? (Should be set by the release plugin to the tag for that release.)

When will the artifacts be available in Maven Central?

In ideal case within one hour, in reality it seems to be few hours, even 12h sometimes.

Summary

We’ve learned how to create a maven repository at oss.sonatype.org, how to sign our artifacts with a private key, how to use the staging repository and finally how to get the released artifacts pushed to the Maven Central repository, where they will be available to all maven users out of the box.

Other useful links

Unrelated but interesting:

Posted in Tools | Tagged: , , , | 1 Comment »

WAS 6.0 ant tasks: Install an app with an external Ant

Posted by Jakub Holý on December 15, 2006

There are special ant taks to install/start/.. an application to a
WebSphere App Server 6.0. It’s easy to run them with WAS’s ant script
(<was>/bin/was_ant.bat) but not so trivial to get them working with an
external ant, which may be necessary because WAS has an old version of ant.
So lets see how to do it.

The important points are:

  • Use IBM JRE
    (or use Sun JRE but setup the environment first by running
    <WAS>/profiles/<your profile>/bin/setupCmdLine in the same
    shell/command line you’ll use to start the ant)
  • Set the environmental variable
    ANT_OPTS=”-Duser.install.root=<WAS>/profiles/<your profile>”
    Strangely, it seems that no other way of passing it to the was ant tasks
    works.
  • Tell Ant where to look for the libraries it needs:
    1. Ant’s own lib/ dir – it must be specified and it must come first
      not to use WAS’ ant
    2. A directory with wsanttasks.jar itself and other jars it depends
      upon (perhaps only wsprofile.jar) – you may specify
      <WAS>/lib
    3. If you want to use scripts in jython, you need to include also
      <WAS>/optionalLibraries

Example:

C:\> set
PATH=C:\development\servers\WebSphere6.0\AppServer\java\jre\bin;%PATH%
C:\> set
JAVA_HOME=C:\development\servers\WebSphere6.0\AppServer\java\jre
C:\> set
ANT_OPTS="-Duser.install.root=C:\development\servers\WebSphere6.0\AppServer/profiles/jh-pokusy"
C:\> c:\apache-ant-1.6.5\bin\ant -verbose -lib ..\apache-ant-1.6.5\lib
-lib c:\development\servers\WebSphere6.0\AppServer\lib -lib
c:\development\servers\WebSphere6.0\AppServer\optionalLibraries -f
myBuild.xml myBuildTarget

myBuild.xml contains:

    <property name="myWas.root"     value="c:\development\servers\WebSphere6.0\AppServer" />

    <taskdef name="wsAdmin"            classname="com.ibm.websphere.ant.tasks.WsAdmin" />
    <taskdef name="wsInstallApp" classname="com.ibm.websphere.ant.tasks.InstallApplication" />
    <taskdef name="wsStartApp" classname="com.ibm.websphere.ant.tasks.StartApplication" />
    <taskdef name="wsUninstallApp" classname="com.ibm.websphere.ant.tasks.UninstallApplication"/>

    <target name="installStartUninstall" depends="_installEarApp,_startEarApp,_uninstallEarApp" />

    <target name="_installEarApp"  description="Deploy the app to WAS 6.0">
                <echo message="Going to deploy '${earToDeploy}' to the server in ${wasroot}..." />
                        <echo message="########## FIRST install ########### ${line.separator}" />
                        <wsInstallApp wasHome="${myWas.root}" conntype="SOAP"
                                      ear="c:\tmp\myApp.ear"
                                      host="localhost" port="8881"
                                      user="me" password="psw"
                                      failonerror="true"
                                      options="-appname MyAppName -verbose true -validateinstall fail -preCompileJSPs true"
                                      />
                        <!-- Note: Anttasks's wsadmin task from was 6.0.0.3 support the attr. profileName="jh-pokusy" -->

                <!-- 1. Ryn my jython script that modifies configuration of the app:
                     (todo: try the atribute: profile="${myWas.root}/bin/setupCmdLine.bat")
                -->
                <wsAdmin wasHome="${myWas.root}" conntype="SOAP" host="localhost" port="8881"
                         host="localhost" port="8881"
                         user="me" password="psw"
                         failonerror="true"
                         script="c:\tmp\configureApplication.jython"
                         lang="jython">
                               <!-- See the script for the arguments it expects -->
                                <arg value="MyAppName"/><!-- app name -->
                </wsAdmin>
        </target>

    <target name="_startEarApp" depends="" description="Start a deployed ear app on the WAS server">
            <wsStartApp wasHome="${myWas.root}" conntype="SOAP"
                        application="MyAppName"
                        host="localhost" port="8881"
                        user="me" password="psw"
                        failonerror="true"
                        />
    </target>

    <target name="_uninstallEarApp" depends="" description="Start a deployed ear app on the WAS server">
            <wsUninstallApp wasHome="${myWas.root}" conntype="SOAP"
                        application="MyAppName"
                        host="localhost" port="8881"
                        user="me" password="psw"
                        failonerror="true"
                        options="-verbose true"
                        />
    </target>

Information

  • host/port for the connection method: see
    <WAS>\profiles\<profile
    name>\properties\wsadmin.properties
  • paths to files (ear, script to execute) – must be absolute!!! – the was
    ant tasks change the working directory => relative paths won’t
    work

Posted in WebSphere | Tagged: , , | 2 Comments »

Redeploy an application (ear/war/…) on JBoss

Posted by Jakub Holý on April 12, 2006

A) Via JMX-console

  1. find jboss.system:service=MainDeployer
  2. invoke listDeployed – find something like:
    org.jboss.deployment.DeploymentInfo@d78e875f { url=file:/C:/jboss-3.2.3/server/default/deploy/tap.ear } 
  3.   copy the url, go back and invoke Redeploy with the url (file:/C:/…) as the parameter.

B) From the command line

The operations that can be invoked via the JBoss JMX console can also be invoked by the cmd line client <jboss home>/bin/twiddle.

C) Manually

Modify application.xml (or web.xml for .war) of the application if deployed in the expload mode (or modify/touch/ the .ear?)

Posted in j2ee | Tagged: , , , | Comments Off