Can't read UTF-8 filenames when launched as an Upstart service

Question

My Java program reads the contents of a directory recursively. This is a sample tree (note the non-ASCII characters): ``` ./sviluppo ./sviluppo/ciaò ./sviluppo/ciaò/subdir ./sviluppo/pippo ./sviluppo/pippo/prova2.txt <-file ./sviluppo/così ``` The program is started as an Upstart service, with a configuration file named like `/init/myservice.conf` ``` description "Private Service" author "AD" start on runlevel [2345] stop on runlevel [! 2345] exec java -jar /home/mainFind.jar >> /tmp/log.txt ``` When I launch the service: ``` root@mdr:/tmp# service myservice start myservice start/running, process 15344 ``` it doesn't log filenames with non-ASCII characters in the name: ``` root@mdr:/tmp# cat /tmp/log.txt Found dir: /mnt/sviluppo/pippo ``` Instead, when I run the command (as root, to mimic what happens when it's started as a service) it works fine, with and without `exec`: ``` root@mdr:/tmp# java -jar /home/mainFind.jar >> /tmp/log.txt root@mdr:/tmp# exec java -jar /home/mainFind.jar >…

Accepted Answer

Java uses a native call to list the contents of a directory. The underlying C runtime relies on the locale concept to build Java `String`s from the byte blob stored by the filesystem as the filename.

When you execute a Java program from a shell (either as a privileged user or an unprivileged one) it carries an environment made of variables. The variable `LANG` is read to transcode the stream of bytes to a Java String, and by default on Ubuntu it's associated to the UTF-8 encoding.

Note that a process need not to be run from any shell, but looking at the code it seems that Upstart is smart enough to understand when the command in the configuration file is intended to be executed from a shell. So, assuming that the JVM is invoked through a shell, the problem is that the variable `LANG` is not set, so the C runtime assumes a default charset, which happens to not be UTF-8. The solution is in the Upstart stanza:

```
description "List UTF-8 encoded filenames"
author "Raffaele Sgarro"
env LANG=en_US.UTF-8
script
  cd /workspace
  java -jar list.jar test > log.txt
end script
```

I used `en_US.UTF-8` as the locale, but any UTF-8 backed one will do just as well. The sources of the test `list.jar`

```
public static void main(String[] args) {
    for (File file : new File(args[0]).listFiles()) {
        System.out.println(file.getName());
    }
}
```

The directory `/workspace/test` contains filenames like `ààà`, `èèè` and so on. Now you can move to the database part ;)

Can't read UTF-8 filenames when launched as an Upstart service

Solution

Problem