Feature #868

Feature #1022: RCP: X.X, proxy settings in TXM and its components

TBX: X.X, R proxy configuration during TXM work session

Ajouté par Matthieu Decorde il y a plus de 4 ans. Mis à jour il y a presque 3 ans.

Statut:Closed Début:17/06/2014
Priorité:Normal Echéance:
Assigné à:- % réalisé:

100%

Catégorie:Network Temps passé: -
Version cible:TXM 0.7.7

Description

R cannot access internet if there is a proxy during the "Check R packages" command. -> Rserve & R configuration

Hypothesis1

R proxy configuration depends on the Operating System:
  • Windows: R uses the "--internet2" option
  • Mac OS X/Linux: R uses the "http_proxy" environment variable

- For Windows: the "--internet2" option can be added automatically. see: http://cran.r-project.org/bin/windows/base/rw-FAQ.html#The-Internet-download-functions-fail_002e
- For Mac OS X and Linux, we have to test the "http_proxy" R env variable is used

Currently R and RServe options are not configured by TXM StatEngine preferences.
(For runtime proxy options see also commands: setInternet2(use=TRUE) and Sys.setenv(http_proxy="http://user::8080/" ))
(We may also use .Rprofile system ? MD: I would rather not do this)

Solution 0

Use .RProfile or .Renviron files to configure the R network.

Solution 1

Match the R network configuration with the OS network configuration
Retrieve the OS proxy configuration: HOW TO (for each OS)

At TXM start up:
  • Windows:
    • run R with: R http_proxy="http_user:http_password@http_proxy:http_port"
    • create the variable before running R: set http_proxy="http_user:http_password@http_proxy:http_port"
    • or use the R command line: Sys.setenv(...). But it doesn't seem to work for this with Windows XP
  • Linux/Mac:
    • use a shell command as: export http_proxy="http_user:http_password@http_proxy:http_port"
    • or use the R command line: Sys.setenv(http_proxy="http_user:http_password@http_proxy:http_port")

Solution 2

Match the R network configuration with the RCP network configuration.
Retrieve the RCP proxy configuration: http://blog.vogella.com/2009/12/08/eclipse-rcp-proxy-preference/

At TXM start up: see Solution 1

Optionnaly, update R proxy when RCP preferences change:
  • Windows: no good solution at this moment (actually one would be to restart R with: R http_proxy="http_user:http_password@http_proxy:http_port") (Sys.setenv() doesn't seems to work for this under Windows XP)
  • Linux/Mac: call in R: Sys.setenv(http_proxy="http_user:http_password@http_proxy:http_port")

Solution 3

Use OS download method instead of the R native download method. The OS download method uses the OS network configuration rather than the R network configuration.

  • in R :
    • set the "download.file.method" R option
    • or use the "method" parameter of the "install.packages" function : install.packages(.., method = "wget")
  • The download method depends on the OS :
    • Linux: wget
    • Mac OS X : curl
    • Windows : no need to install packages

NOTE: Linux "export" and Windows "set" command create the variable in the local session scope, they are not permanently stored.

Validation tests

  • run TXM with a proxy connection
  • enable all log levels in preferences and enable "Run RServe in debug mode"
  • restart TXM
  • run a .R file containing
install.packages("textometry", dependencies=TRUE, repos="http://cran.rstudio.com/");

=> in the console you should have some entries like this:

Rserve>Calling R_tryEval for expression 1 [type=6] ...
Rserve>Installing package into 'C:/Users/s/TXM/R/libraries'
Rserve>(as 'lib' is unspecified)
Rserve>trying URL 'http://cran.rstudio.com/bin/windows/contrib/3.1/textometry_0.1.4.zip'
Rserve>Content type 'application/zip' length 38040 bytes (37 Kb)
Rserve>opened URL
Rserve>downloaded 37 Kb
Rserve>
Rserve>package 'textometry' successfully unpacked and MD5 sums checked
Rserve>
Rserve>The downloaded binary packages are in
Rserve>    C:\Users\s\AppData\Local\Temp\RtmpCOyMKe\downloaded_packages
  • the test has failed if you have this line in the console instead of the above log:
Warning: unable to access index for repository http://cran.rstudio.com/src/contrib

Demandes liées

lié à Bug #525: RCP: 0.7.5, Fix timeout error at start-up when user is us... Closed 17/01/2014
lié à Bug #857: RCP: 0.7.6, the check and update R packages command hangs... New 12/06/2014

Historique

#1 Mis à jour par Matthieu Decorde il y a plus de 4 ans

  • Description mis à jour (diff)

#2 Mis à jour par Matthieu Decorde il y a plus de 4 ans

  • Description mis à jour (diff)

#3 Mis à jour par Serge Heiden il y a plus de 4 ans

  • Description mis à jour (diff)

#4 Mis à jour par Matthieu Decorde il y a plus de 4 ans

  • % réalisé changé de 0 à 80

#5 Mis à jour par Matthieu Decorde il y a plus de 4 ans

  • Sujet changé de TBX: X.X, R configuration à TBX: X.X, R proxy configuration

#6 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

  • Version cible changé de TXM 0.7.6 à TXM 0.7.7

#7 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

Adding "--internet2" in "R args" and/or "Rserve args" parameters of the RCP preference doesn't fix the problem (tested on Windows XP)

#8 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

About previous note, if I remember well our tests, the problem is that the generated command line by TXM to run R/Rserve contains some double spaces and/or two times the string "--internet2", need to recheck this.

#9 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

  • Description mis à jour (diff)

After some research we may use:

http_proxy="http://user:password@proxy.example.com:8080/" 

argument in the R starting command line. It may works on the 3 OS, I tested on Windows XP under proxy with success.
If it works well we may get the Eclipse Network Connection preferences (proxy, user, pass, port) from RCP store then compute the command line from these variables. (It means we need to create these preferences in TBX too)

#10 Mis à jour par Matthieu Decorde il y a plus de 4 ans

  • Tâche parente mis à #1022

#11 Mis à jour par Matthieu Decorde il y a plus de 4 ans

  • % réalisé changé de 80 à 20

#12 Mis à jour par Matthieu Decorde il y a plus de 4 ans

  • Description mis à jour (diff)

#13 Mis à jour par Matthieu Decorde il y a plus de 4 ans

  • Description mis à jour (diff)

#14 Mis à jour par Matthieu Decorde il y a plus de 4 ans

  • Description mis à jour (diff)

#15 Mis à jour par Matthieu Decorde il y a plus de 4 ans

  • Description mis à jour (diff)

#16 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

Some new tests:
- Ubuntu Network manager seems to create the environment variables "http_proxy", "https_proxy", etc. when setting some proxies within its GUI.
- Mac OS X Mavericke does NOT create these variables. First proposal is to get the proxy configuration with:

networksetup -getwebproxy Ethernet

The problem here is that we need to list all the active network interfaces/services ? (wireless, etc. ?)

The argument "http_proxy" is not usable in command line under Ubuntu or Mac OS X, it only works under Windows.

Unfortunately the runtime command :

Sys.setenv(http_proxy="http://user:password@proxy.example.com:8080");

does not work under Windows.
We still can use "--internet2" but I guess for R in TXM, the better feature would be to match the RCP proxy configuration and not the OS proxy configuration ?

(Another solution for the setup could be to ask to user.)

#17 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

  • Description mis à jour (diff)

#18 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

  • Description mis à jour (diff)

#19 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

  • Description mis à jour (diff)

#20 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

  • Description mis à jour (diff)

#21 Mis à jour par Matthieu Decorde il y a plus de 4 ans

  • Description mis à jour (diff)

#22 Mis à jour par Matthieu Decorde il y a plus de 4 ans

  • Version cible changé de TXM 0.7.7 à TXM 0.7.6

#23 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

  • Version cible changé de TXM 0.7.6 à TXM 0.7.7

#24 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

  • Catégorie changé de Stats / R à Network

#25 Mis à jour par Matthieu Decorde il y a plus de 4 ans

  • Sujet changé de TBX: X.X, R proxy configuration à TBX: X.X, R proxy configuration during TXM work session
  • Description mis à jour (diff)

#26 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

Important note: for Mac, after new tests we need to do:

Sys.setenv(http_proxy="http://user:password@proxy.example.com:8080");
install.packages("textometry", method = "curl")

#27 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

To detect proxy with Java: http://stackoverflow.com/questions/376101/setting-jvm-jre-to-use-windows-proxy-automatically

System.setProperty("java.net.useSystemProxies", "true");
System.out.println("detecting proxies");
List l = null;
try {
    l = ProxySelector.getDefault().select(new URI("http://foo/bar"));
} 
catch (URISyntaxException e) {
    e.printStackTrace();
}
if (l != null) {
    for (Iterator iter = l.iterator(); iter.hasNext();) {
        java.net.Proxy proxy = (java.net.Proxy) iter.next();
        System.out.println("proxy hostname : " + proxy.type());

        InetSocketAddress addr = (InetSocketAddress) proxy.address();

        if (addr == null) {
            System.out.println("No Proxy");
        } else {
            System.out.println("proxy hostname : " + addr.getHostName());
            System.setProperty("http.proxyHost", addr.getHostName());
            System.out.println("proxy port : " + addr.getPort());
            System.setProperty("http.proxyPort", Integer.toString(addr.getPort()));
        }
    }
}

Also see:

System.getProperty("http.proxyHost");
System.getProperty("http.proxyPort");

#28 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

We may execute this to retrieve proxies on Mac:

sudo networksetup -getwebproxy Airport | awk {'print $2'} | awk {'getline l2; getline l3; print "http://"l2":"l3'} | head -n 1
sudo networksetup -getwebproxy Ethernet| awk {'print $2'} | awk {'getline l2; getline l3; print "http://"l2":"l3'} | head -n 1

returns:

http://:
http://proxy.univ-xx.fr:3128

with my Ethernet connection. Then give it to Sys.setenv() in R.

#29 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

A cleaner way would be to loop through the results of:

networksetup -listallnetworkservices

but it seems hard to get the user name and password.

Also see for .pac:

networksetup -getautoproxyurl

#30 Mis à jour par Sebastien Jacquot il y a plus de 4 ans

New tests for Windows, this works:
run R http_proxy="http://proxy.xx.fr:3128"
NOTE: it's an argument, not an option so we musn't put "--"

Sys.setenv(http_proxy="http://proxy.xx.fr:3128")

finally seems to work too.

#31 Mis à jour par Matthieu Decorde il y a environ 4 ans

  • % réalisé changé de 20 à 60

Linux and Windows tests passed

#32 Mis à jour par Sebastien Jacquot il y a presque 4 ans

  • Statut changé de New à Closed
  • % réalisé changé de 60 à 100

The solution is not fully satisfactory.
We need to match the R network configuration to the RCP network configuration.
See #1336.

#33 Mis à jour par Sebastien Jacquot il y a plus de 3 ans

Implementing the solution 1 means changing the configuration needs a full restart of R/Rserve.
Implementing the solution 3 means all the network commands in a R console won't be directly available for proxy users.

#34 Mis à jour par Sebastien Jacquot il y a plus de 3 ans

  • Description mis à jour (diff)
  • Statut changé de Closed à In Progress
  • % réalisé changé de 100 à 80

#35 Mis à jour par Sebastien Jacquot il y a plus de 3 ans

At first we need to reduce the connection time out in R. See getOption("timeout"). Default seems to be 60 seconds in Linux.

#36 Mis à jour par Sebastien Jacquot il y a plus de 3 ans

To reduce R connection time out:

options(timeout = x);

with x in number of seconds.

TMP: for muli-target tests purpose:

print(Sys.getenv("http_proxy"));
Sys.setenv(http_proxy="http://proxy-web.univ-fcomte.fr:3128");
print(Sys.getenv("http_proxy"));
options(timeout = 1);
print(getOption("timeout"));
install.packages("textometry", dependencies=TRUE, repos="http://cran.rstudio.com/");

#37 Mis à jour par Sebastien Jacquot il y a plus de 3 ans

  • Description mis à jour (diff)

#38 Mis à jour par Sebastien Jacquot il y a plus de 3 ans

Tests with last setup 0.7.7b2.2 Win and Linux:

print(Sys.getenv("http_proxy"));

returns well the proxy on Win and Linux.

install.packages("textometry", dependencies=TRUE, repos="http://cran.rstudio.com/");

freezes the UI on Win 7, Win XP and Linux.
The strange thing here is that on the Win XP machine, I have a firewall which asks me to authorize the connection to http://cran.rstudio.com/ when I type this line, but after the connection is accepted, the UI freezes until the R time out (about 30 minutes).

Log after the UI freeze is:

Warning: unable to access index for repository http://cran.rstudio.com/src/contrib

It seems there is some traffic on Linux, tcpdump log:

19:14:08.635729 IP ubuntu.local.6951 > 192.168.88.2.domain: 17681+ A? cran.rstudio.com. (34)
19:14:08.637274 IP 192.168.88.2.domain > ubuntu.local.6951: 17681 9/4/4 CNAME dqhc1zxg5b5jh.cloudfront.net., A 54.230.218.108, A 54.230.218.234, A 54.192.219.174, A 54.192.218.249, A 54.192.219.234, A 54.230.218.218, A 54.192.218.182, A 54.192.219.197 (402)
19:14:08.637421 IP ubuntu.local.34305 > server-54-230-218-108.mrs50.r.cloudfront.net.http: Flags [S], seq 3920735817, win 29200, options [mss 1460,sackOK,TS val 19121861 ecr 0,nop,wscale 7], length 0
19:14:09.612213 ARP, Request who-has 192.168.88.162 tell 192.168.88.2, length 46
19:14:09.635113 IP ubuntu.local.34305 > server-54-230-218-108.mrs50.r.cloudfront.net.http: Flags [S], seq 3920735817, win 29200, options [mss 1460,sackOK,TS val 19122111 ecr 0,nop,wscale 7], length 0
19:14:09.639198 IP ubuntu.local.54203 > server-54-230-218-234.mrs50.r.cloudfront.net.http: Flags [S], seq 1808043822, win 29200, options [mss 1460,sackOK,TS val 19122112 ecr 0,nop,wscale 7], length 0
...

#39 Mis à jour par Sebastien Jacquot il y a plus de 3 ans

It seems:

install.packages("textometry", dependencies=TRUE, repos="http://cran.rstudio.com");

works on Linux when calling from R in a terminal but not in TXM therefore through RServe.

Useful option for debugging to change network communications log level?

options(internet.info = 0)

#40 Mis à jour par Sebastien Jacquot il y a plus de 3 ans

  • Statut changé de In Progress à Feedback
Fixed, here are the steps of the last process state:
  • TBX tries to get the HTTP system proxy configuration from the RCP plugin org.eclipse.core.net native implementations and from the Java system properties if the first solution retrieved nothing
  • R is launched without proxy setting
  • after started, R time out is reduced from 60 seconds to 3 through RServe by calling 'options(timeout = 3);'
  • after started, R HTTP proxy is configured through RServe by calling 'Sys.setenv(http_proxy="http://<user>:<password>@<host>:<port>")' using values of the retrieved system proxy settings

#41 Mis à jour par Sebastien Jacquot il y a plus de 3 ans

  • Description mis à jour (diff)

#42 Mis à jour par Sebastien Jacquot il y a plus de 3 ans

Successfully validated with system proxy configurations on Win 7 64bit, Win XP 32bit, Ubuntu 14.04 (VM) and Mac OS X 10.9.4 (VM).

#43 Mis à jour par Sebastien Jacquot il y a presque 3 ans

  • Statut changé de Feedback à Closed
  • % réalisé changé de 80 à 100

Formats disponibles : Atom PDF