Feature #868

Feature #1022: RCP: X.X, proxy settings in TXM and its components

TBX: X.X, R proxy configuration during TXM work session

Added by Matthieu Decorde almost 5 years ago. Updated about 3 years ago.

Status:Closed Start date:06/17/2014
Priority:Normal Due date:
Assignee:- % Done:

100%

Category:Network Spent time: -
Target version:TXM 0.7.7

Description

R cannot access internet if there is a proxy during the "Check R packages" command. -> Rserve & R configuration

Hypothesis1

R proxy configuration depends on the Operating System:
  • Windows: R uses the "--internet2" option
  • Mac OS X/Linux: R uses the "http_proxy" environment variable

- For Windows: the "--internet2" option can be added automatically. see: http://cran.r-project.org/bin/windows/base/rw-FAQ.html#The-Internet-download-functions-fail_002e
- For Mac OS X and Linux, we have to test the "http_proxy" R env variable is used

Currently R and RServe options are not configured by TXM StatEngine preferences.
(For runtime proxy options see also commands: setInternet2(use=TRUE) and Sys.setenv(http_proxy="http://user::8080/" ))
(We may also use .Rprofile system ? MD: I would rather not do this)

Solution 0

Use .RProfile or .Renviron files to configure the R network.

Solution 1

Match the R network configuration with the OS network configuration
Retrieve the OS proxy configuration: HOW TO (for each OS)

At TXM start up:
  • Windows:
    • run R with: R http_proxy="http_user:http_password@http_proxy:http_port"
    • create the variable before running R: set http_proxy="http_user:http_password@http_proxy:http_port"
    • or use the R command line: Sys.setenv(...). But it doesn't seem to work for this with Windows XP
  • Linux/Mac:
    • use a shell command as: export http_proxy="http_user:http_password@http_proxy:http_port"
    • or use the R command line: Sys.setenv(http_proxy="http_user:http_password@http_proxy:http_port")

Solution 2

Match the R network configuration with the RCP network configuration.
Retrieve the RCP proxy configuration: http://blog.vogella.com/2009/12/08/eclipse-rcp-proxy-preference/

At TXM start up: see Solution 1

Optionnaly, update R proxy when RCP preferences change:
  • Windows: no good solution at this moment (actually one would be to restart R with: R http_proxy="http_user:http_password@http_proxy:http_port") (Sys.setenv() doesn't seems to work for this under Windows XP)
  • Linux/Mac: call in R: Sys.setenv(http_proxy="http_user:http_password@http_proxy:http_port")

Solution 3

Use OS download method instead of the R native download method. The OS download method uses the OS network configuration rather than the R network configuration.

  • in R :
    • set the "download.file.method" R option
    • or use the "method" parameter of the "install.packages" function : install.packages(.., method = "wget")
  • The download method depends on the OS :
    • Linux: wget
    • Mac OS X : curl
    • Windows : no need to install packages

NOTE: Linux "export" and Windows "set" command create the variable in the local session scope, they are not permanently stored.

Validation tests

  • run TXM with a proxy connection
  • enable all log levels in preferences and enable "Run RServe in debug mode"
  • restart TXM
  • run a .R file containing
install.packages("textometry", dependencies=TRUE, repos="http://cran.rstudio.com/");

=> in the console you should have some entries like this:

Rserve>Calling R_tryEval for expression 1 [type=6] ...
Rserve>Installing package into 'C:/Users/s/TXM/R/libraries'
Rserve>(as 'lib' is unspecified)
Rserve>trying URL 'http://cran.rstudio.com/bin/windows/contrib/3.1/textometry_0.1.4.zip'
Rserve>Content type 'application/zip' length 38040 bytes (37 Kb)
Rserve>opened URL
Rserve>downloaded 37 Kb
Rserve>
Rserve>package 'textometry' successfully unpacked and MD5 sums checked
Rserve>
Rserve>The downloaded binary packages are in
Rserve>    C:\Users\s\AppData\Local\Temp\RtmpCOyMKe\downloaded_packages
  • the test has failed if you have this line in the console instead of the above log:
Warning: unable to access index for repository http://cran.rstudio.com/src/contrib

Related issues

related to Bug #525: RCP: 0.7.5, Fix timeout error at start-up when user is us... Closed 01/17/2014
related to Bug #857: RCP: 0.7.6, the check and update R packages command hangs... New 06/12/2014

History

#1 Updated by Matthieu Decorde almost 5 years ago

  • Description updated (diff)

#2 Updated by Matthieu Decorde almost 5 years ago

  • Description updated (diff)

#3 Updated by Serge Heiden almost 5 years ago

  • Description updated (diff)

#4 Updated by Matthieu Decorde almost 5 years ago

  • % Done changed from 0 to 80

#5 Updated by Matthieu Decorde almost 5 years ago

  • Subject changed from TBX: X.X, R configuration to TBX: X.X, R proxy configuration

#6 Updated by Sebastien Jacquot over 4 years ago

  • Target version changed from TXM 0.7.6 to TXM 0.7.7

#7 Updated by Sebastien Jacquot over 4 years ago

Adding "--internet2" in "R args" and/or "Rserve args" parameters of the RCP preference doesn't fix the problem (tested on Windows XP)

#8 Updated by Sebastien Jacquot over 4 years ago

About previous note, if I remember well our tests, the problem is that the generated command line by TXM to run R/Rserve contains some double spaces and/or two times the string "--internet2", need to recheck this.

#9 Updated by Sebastien Jacquot over 4 years ago

  • Description updated (diff)

After some research we may use:

http_proxy="http://user:password@proxy.example.com:8080/" 

argument in the R starting command line. It may works on the 3 OS, I tested on Windows XP under proxy with success.
If it works well we may get the Eclipse Network Connection preferences (proxy, user, pass, port) from RCP store then compute the command line from these variables. (It means we need to create these preferences in TBX too)

#10 Updated by Matthieu Decorde over 4 years ago

  • Parent task set to #1022

#11 Updated by Matthieu Decorde over 4 years ago

  • % Done changed from 80 to 20

#12 Updated by Matthieu Decorde over 4 years ago

  • Description updated (diff)

#13 Updated by Matthieu Decorde over 4 years ago

  • Description updated (diff)

#14 Updated by Matthieu Decorde over 4 years ago

  • Description updated (diff)

#15 Updated by Matthieu Decorde over 4 years ago

  • Description updated (diff)

#16 Updated by Sebastien Jacquot over 4 years ago

Some new tests:
- Ubuntu Network manager seems to create the environment variables "http_proxy", "https_proxy", etc. when setting some proxies within its GUI.
- Mac OS X Mavericke does NOT create these variables. First proposal is to get the proxy configuration with:

networksetup -getwebproxy Ethernet

The problem here is that we need to list all the active network interfaces/services ? (wireless, etc. ?)

The argument "http_proxy" is not usable in command line under Ubuntu or Mac OS X, it only works under Windows.

Unfortunately the runtime command :

Sys.setenv(http_proxy="http://user:password@proxy.example.com:8080");

does not work under Windows.
We still can use "--internet2" but I guess for R in TXM, the better feature would be to match the RCP proxy configuration and not the OS proxy configuration ?

(Another solution for the setup could be to ask to user.)

#17 Updated by Sebastien Jacquot over 4 years ago

  • Description updated (diff)

#18 Updated by Sebastien Jacquot over 4 years ago

  • Description updated (diff)

#19 Updated by Sebastien Jacquot over 4 years ago

  • Description updated (diff)

#20 Updated by Sebastien Jacquot over 4 years ago

  • Description updated (diff)

#21 Updated by Matthieu Decorde over 4 years ago

  • Description updated (diff)

#22 Updated by Matthieu Decorde over 4 years ago

  • Target version changed from TXM 0.7.7 to TXM 0.7.6

#23 Updated by Sebastien Jacquot over 4 years ago

  • Target version changed from TXM 0.7.6 to TXM 0.7.7

#24 Updated by Sebastien Jacquot over 4 years ago

  • Category changed from Stats / R to Network

#25 Updated by Matthieu Decorde over 4 years ago

  • Subject changed from TBX: X.X, R proxy configuration to TBX: X.X, R proxy configuration during TXM work session
  • Description updated (diff)

#26 Updated by Sebastien Jacquot over 4 years ago

Important note: for Mac, after new tests we need to do:

Sys.setenv(http_proxy="http://user:password@proxy.example.com:8080");
install.packages("textometry", method = "curl")

#27 Updated by Sebastien Jacquot over 4 years ago

To detect proxy with Java: http://stackoverflow.com/questions/376101/setting-jvm-jre-to-use-windows-proxy-automatically

System.setProperty("java.net.useSystemProxies", "true");
System.out.println("detecting proxies");
List l = null;
try {
    l = ProxySelector.getDefault().select(new URI("http://foo/bar"));
} 
catch (URISyntaxException e) {
    e.printStackTrace();
}
if (l != null) {
    for (Iterator iter = l.iterator(); iter.hasNext();) {
        java.net.Proxy proxy = (java.net.Proxy) iter.next();
        System.out.println("proxy hostname : " + proxy.type());

        InetSocketAddress addr = (InetSocketAddress) proxy.address();

        if (addr == null) {
            System.out.println("No Proxy");
        } else {
            System.out.println("proxy hostname : " + addr.getHostName());
            System.setProperty("http.proxyHost", addr.getHostName());
            System.out.println("proxy port : " + addr.getPort());
            System.setProperty("http.proxyPort", Integer.toString(addr.getPort()));
        }
    }
}

Also see:

System.getProperty("http.proxyHost");
System.getProperty("http.proxyPort");

#28 Updated by Sebastien Jacquot over 4 years ago

We may execute this to retrieve proxies on Mac:

sudo networksetup -getwebproxy Airport | awk {'print $2'} | awk {'getline l2; getline l3; print "http://"l2":"l3'} | head -n 1
sudo networksetup -getwebproxy Ethernet| awk {'print $2'} | awk {'getline l2; getline l3; print "http://"l2":"l3'} | head -n 1

returns:

http://:
http://proxy.univ-xx.fr:3128

with my Ethernet connection. Then give it to Sys.setenv() in R.

#29 Updated by Sebastien Jacquot over 4 years ago

A cleaner way would be to loop through the results of:

networksetup -listallnetworkservices

but it seems hard to get the user name and password.

Also see for .pac:

networksetup -getautoproxyurl

#30 Updated by Sebastien Jacquot over 4 years ago

New tests for Windows, this works:
run R http_proxy="http://proxy.xx.fr:3128"
NOTE: it's an argument, not an option so we musn't put "--"

Sys.setenv(http_proxy="http://proxy.xx.fr:3128")

finally seems to work too.

#31 Updated by Matthieu Decorde over 4 years ago

  • % Done changed from 20 to 60

Linux and Windows tests passed

#32 Updated by Sebastien Jacquot almost 4 years ago

  • Status changed from New to Closed
  • % Done changed from 60 to 100

The solution is not fully satisfactory.
We need to match the R network configuration to the RCP network configuration.
See #1336.

#33 Updated by Sebastien Jacquot almost 4 years ago

Implementing the solution 1 means changing the configuration needs a full restart of R/Rserve.
Implementing the solution 3 means all the network commands in a R console won't be directly available for proxy users.

#34 Updated by Sebastien Jacquot almost 4 years ago

  • Description updated (diff)
  • Status changed from Closed to In Progress
  • % Done changed from 100 to 80

#35 Updated by Sebastien Jacquot almost 4 years ago

At first we need to reduce the connection time out in R. See getOption("timeout"). Default seems to be 60 seconds in Linux.

#36 Updated by Sebastien Jacquot almost 4 years ago

To reduce R connection time out:

options(timeout = x);

with x in number of seconds.

TMP: for muli-target tests purpose:

print(Sys.getenv("http_proxy"));
Sys.setenv(http_proxy="http://proxy-web.univ-fcomte.fr:3128");
print(Sys.getenv("http_proxy"));
options(timeout = 1);
print(getOption("timeout"));
install.packages("textometry", dependencies=TRUE, repos="http://cran.rstudio.com/");

#37 Updated by Sebastien Jacquot almost 4 years ago

  • Description updated (diff)

#38 Updated by Sebastien Jacquot almost 4 years ago

Tests with last setup 0.7.7b2.2 Win and Linux:

print(Sys.getenv("http_proxy"));

returns well the proxy on Win and Linux.

install.packages("textometry", dependencies=TRUE, repos="http://cran.rstudio.com/");

freezes the UI on Win 7, Win XP and Linux.
The strange thing here is that on the Win XP machine, I have a firewall which asks me to authorize the connection to http://cran.rstudio.com/ when I type this line, but after the connection is accepted, the UI freezes until the R time out (about 30 minutes).

Log after the UI freeze is:

Warning: unable to access index for repository http://cran.rstudio.com/src/contrib

It seems there is some traffic on Linux, tcpdump log:

19:14:08.635729 IP ubuntu.local.6951 > 192.168.88.2.domain: 17681+ A? cran.rstudio.com. (34)
19:14:08.637274 IP 192.168.88.2.domain > ubuntu.local.6951: 17681 9/4/4 CNAME dqhc1zxg5b5jh.cloudfront.net., A 54.230.218.108, A 54.230.218.234, A 54.192.219.174, A 54.192.218.249, A 54.192.219.234, A 54.230.218.218, A 54.192.218.182, A 54.192.219.197 (402)
19:14:08.637421 IP ubuntu.local.34305 > server-54-230-218-108.mrs50.r.cloudfront.net.http: Flags [S], seq 3920735817, win 29200, options [mss 1460,sackOK,TS val 19121861 ecr 0,nop,wscale 7], length 0
19:14:09.612213 ARP, Request who-has 192.168.88.162 tell 192.168.88.2, length 46
19:14:09.635113 IP ubuntu.local.34305 > server-54-230-218-108.mrs50.r.cloudfront.net.http: Flags [S], seq 3920735817, win 29200, options [mss 1460,sackOK,TS val 19122111 ecr 0,nop,wscale 7], length 0
19:14:09.639198 IP ubuntu.local.54203 > server-54-230-218-234.mrs50.r.cloudfront.net.http: Flags [S], seq 1808043822, win 29200, options [mss 1460,sackOK,TS val 19122112 ecr 0,nop,wscale 7], length 0
...

#39 Updated by Sebastien Jacquot almost 4 years ago

It seems:

install.packages("textometry", dependencies=TRUE, repos="http://cran.rstudio.com");

works on Linux when calling from R in a terminal but not in TXM therefore through RServe.

Useful option for debugging to change network communications log level?

options(internet.info = 0)

#40 Updated by Sebastien Jacquot almost 4 years ago

  • Status changed from In Progress to Feedback
Fixed, here are the steps of the last process state:
  • TBX tries to get the HTTP system proxy configuration from the RCP plugin org.eclipse.core.net native implementations and from the Java system properties if the first solution retrieved nothing
  • R is launched without proxy setting
  • after started, R time out is reduced from 60 seconds to 3 through RServe by calling 'options(timeout = 3);'
  • after started, R HTTP proxy is configured through RServe by calling 'Sys.setenv(http_proxy="http://<user>:<password>@<host>:<port>")' using values of the retrieved system proxy settings

#41 Updated by Sebastien Jacquot almost 4 years ago

  • Description updated (diff)

#42 Updated by Sebastien Jacquot almost 4 years ago

Successfully validated with system proxy configurations on Win 7 64bit, Win XP 32bit, Ubuntu 14.04 (VM) and Mac OS X 10.9.4 (VM).

#43 Updated by Sebastien Jacquot about 3 years ago

  • Status changed from Feedback to Closed
  • % Done changed from 80 to 100

Also available in: Atom PDF