Feature #868
Feature #1022: RCP: X.X, proxy settings in TXM and its components
TBX: X.X, R proxy configuration during TXM work session
Statut: | Closed | Début: | 17/06/2014 | |
---|---|---|---|---|
Priorité: | Normal | Echéance: | ||
Assigné à: | - | % réalisé: | 100% |
|
Catégorie: | Network | Temps passé: | - | |
Version cible: | TXM 0.7.7 |
Description
R cannot access internet if there is a proxy during the "Check R packages" command. -> Rserve & R configuration
Hypothesis1¶
R proxy configuration depends on the Operating System:- Windows: R uses the "--internet2" option
- Mac OS X/Linux: R uses the "http_proxy" environment variable
- For Windows: the "--internet2" option can be added automatically. see: http://cran.r-project.org/bin/windows/base/rw-FAQ.html#The-Internet-download-functions-fail_002e
- For Mac OS X and Linux, we have to test the "http_proxy" R env variable is used
Currently R and RServe options are not configured by TXM StatEngine preferences.
(For runtime proxy options see also commands: setInternet2(use=TRUE) and Sys.setenv(http_proxy="http://user:password@proxy.example.com:8080/" ))
(We may also use .Rprofile system ? MD: I would rather not do this)
Solution 0¶
Use .RProfile or .Renviron files to configure the R network.
Solution 1¶
Match the R network configuration with the OS network configuration
Retrieve the OS proxy configuration: HOW TO (for each OS)
- Windows:
- run R with: R http_proxy="http_user:http_password@http_proxy:http_port"
- create the variable before running R: set http_proxy="http_user:http_password@http_proxy:http_port"
- or use the R command line: Sys.setenv(...). But it doesn't seem to work for this with Windows XP
- Linux/Mac:
- use a shell command as: export http_proxy="http_user:http_password@http_proxy:http_port"
- or use the R command line: Sys.setenv(http_proxy="http_user:http_password@http_proxy:http_port")
Solution 2¶
Match the R network configuration with the RCP network configuration.
Retrieve the RCP proxy configuration: http://blog.vogella.com/2009/12/08/eclipse-rcp-proxy-preference/
At TXM start up: see Solution 1
Optionnaly, update R proxy when RCP preferences change:- Windows: no good solution at this moment (actually one would be to restart R with: R http_proxy="http_user:http_password@http_proxy:http_port") (Sys.setenv() doesn't seems to work for this under Windows XP)
- Linux/Mac: call in R: Sys.setenv(http_proxy="http_user:http_password@http_proxy:http_port")
Solution 3¶
Use OS download method instead of the R native download method. The OS download method uses the OS network configuration rather than the R network configuration.
- in R :
- set the "download.file.method" R option
- or use the "method" parameter of the "install.packages" function : install.packages(.., method = "wget")
- The download method depends on the OS :
- Linux: wget
- Mac OS X : curl
- Windows : no need to install packages
NOTE: Linux "export" and Windows "set" command create the variable in the local session scope, they are not permanently stored.
Validation tests¶
- run TXM with a proxy connection
- enable all log levels in preferences and enable "Run RServe in debug mode"
- restart TXM
- run a .R file containing
install.packages("textometry", dependencies=TRUE, repos="http://cran.rstudio.com/");
=> in the console you should have some entries like this:
Rserve>Calling R_tryEval for expression 1 [type=6] ... Rserve>Installing package into 'C:/Users/s/TXM/R/libraries' Rserve>(as 'lib' is unspecified) Rserve>trying URL 'http://cran.rstudio.com/bin/windows/contrib/3.1/textometry_0.1.4.zip' Rserve>Content type 'application/zip' length 38040 bytes (37 Kb) Rserve>opened URL Rserve>downloaded 37 Kb Rserve> Rserve>package 'textometry' successfully unpacked and MD5 sums checked Rserve> Rserve>The downloaded binary packages are in Rserve> C:\Users\s\AppData\Local\Temp\RtmpCOyMKe\downloaded_packages
- the test has failed if you have this line in the console instead of the above log:
Warning: unable to access index for repository http://cran.rstudio.com/src/contrib
Demandes liées
Historique
#1 Mis à jour par Matthieu Decorde il y a plus de 4 ans
- Description mis à jour (diff)
#2 Mis à jour par Matthieu Decorde il y a plus de 4 ans
- Description mis à jour (diff)
#3 Mis à jour par Serge Heiden il y a plus de 4 ans
- Description mis à jour (diff)
#4 Mis à jour par Matthieu Decorde il y a plus de 4 ans
- % réalisé changé de 0 à 80
#5 Mis à jour par Matthieu Decorde il y a plus de 4 ans
- Sujet changé de TBX: X.X, R configuration à TBX: X.X, R proxy configuration
#6 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
- Version cible changé de TXM 0.7.6 à TXM 0.7.7
#7 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
Adding "--internet2" in "R args" and/or "Rserve args" parameters of the RCP preference doesn't fix the problem (tested on Windows XP)
#8 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
About previous note, if I remember well our tests, the problem is that the generated command line by TXM to run R/Rserve contains some double spaces and/or two times the string "--internet2", need to recheck this.
#9 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
- Description mis à jour (diff)
After some research we may use:
http_proxy="http://user:password@proxy.example.com:8080/"
argument in the R starting command line. It may works on the 3 OS, I tested on Windows XP under proxy with success.
If it works well we may get the Eclipse Network Connection preferences (proxy, user, pass, port) from RCP store then compute the command line from these variables. (It means we need to create these preferences in TBX too)
#10 Mis à jour par Matthieu Decorde il y a plus de 4 ans
- Tâche parente mis à #1022
#11 Mis à jour par Matthieu Decorde il y a plus de 4 ans
- % réalisé changé de 80 à 20
#12 Mis à jour par Matthieu Decorde il y a plus de 4 ans
- Description mis à jour (diff)
#13 Mis à jour par Matthieu Decorde il y a plus de 4 ans
- Description mis à jour (diff)
#14 Mis à jour par Matthieu Decorde il y a plus de 4 ans
- Description mis à jour (diff)
#15 Mis à jour par Matthieu Decorde il y a plus de 4 ans
- Description mis à jour (diff)
#16 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
Some new tests:
- Ubuntu Network manager seems to create the environment variables "http_proxy", "https_proxy", etc. when setting some proxies within its GUI.
- Mac OS X Mavericke does NOT create these variables. First proposal is to get the proxy configuration with:
networksetup -getwebproxy Ethernet
The problem here is that we need to list all the active network interfaces/services ? (wireless, etc. ?)
The argument "http_proxy" is not usable in command line under Ubuntu or Mac OS X, it only works under Windows.
Unfortunately the runtime command :
Sys.setenv(http_proxy="http://user:password@proxy.example.com:8080");
does not work under Windows.
We still can use "--internet2" but I guess for R in TXM, the better feature would be to match the RCP proxy configuration and not the OS proxy configuration ?
(Another solution for the setup could be to ask to user.)
#17 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
- Description mis à jour (diff)
#18 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
- Description mis à jour (diff)
#19 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
- Description mis à jour (diff)
#20 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
- Description mis à jour (diff)
#21 Mis à jour par Matthieu Decorde il y a plus de 4 ans
- Description mis à jour (diff)
#22 Mis à jour par Matthieu Decorde il y a plus de 4 ans
- Version cible changé de TXM 0.7.7 à TXM 0.7.6
#23 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
- Version cible changé de TXM 0.7.6 à TXM 0.7.7
#24 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
- Catégorie changé de Stats / R à Network
#25 Mis à jour par Matthieu Decorde il y a plus de 4 ans
- Sujet changé de TBX: X.X, R proxy configuration à TBX: X.X, R proxy configuration during TXM work session
- Description mis à jour (diff)
#26 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
Important note: for Mac, after new tests we need to do:
Sys.setenv(http_proxy="http://user:password@proxy.example.com:8080"); install.packages("textometry", method = "curl")
#27 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
To detect proxy with Java: http://stackoverflow.com/questions/376101/setting-jvm-jre-to-use-windows-proxy-automatically
System.setProperty("java.net.useSystemProxies", "true"); System.out.println("detecting proxies"); List l = null; try { l = ProxySelector.getDefault().select(new URI("http://foo/bar")); } catch (URISyntaxException e) { e.printStackTrace(); } if (l != null) { for (Iterator iter = l.iterator(); iter.hasNext();) { java.net.Proxy proxy = (java.net.Proxy) iter.next(); System.out.println("proxy hostname : " + proxy.type()); InetSocketAddress addr = (InetSocketAddress) proxy.address(); if (addr == null) { System.out.println("No Proxy"); } else { System.out.println("proxy hostname : " + addr.getHostName()); System.setProperty("http.proxyHost", addr.getHostName()); System.out.println("proxy port : " + addr.getPort()); System.setProperty("http.proxyPort", Integer.toString(addr.getPort())); } } }
Also see:
System.getProperty("http.proxyHost"); System.getProperty("http.proxyPort");
#28 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
We may execute this to retrieve proxies on Mac:
sudo networksetup -getwebproxy Airport | awk {'print $2'} | awk {'getline l2; getline l3; print "http://"l2":"l3'} | head -n 1
sudo networksetup -getwebproxy Ethernet| awk {'print $2'} | awk {'getline l2; getline l3; print "http://"l2":"l3'} | head -n 1
returns:
http://:
http://proxy.univ-xx.fr:3128
with my Ethernet connection. Then give it to Sys.setenv() in R.
#29 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
A cleaner way would be to loop through the results of:
networksetup -listallnetworkservices
but it seems hard to get the user name and password.
Also see for .pac:
networksetup -getautoproxyurl
#30 Mis à jour par Sebastien Jacquot il y a plus de 4 ans
New tests for Windows, this works:
run R http_proxy="http://proxy.xx.fr:3128"
NOTE: it's an argument, not an option so we musn't put "--"
Sys.setenv(http_proxy="http://proxy.xx.fr:3128")
finally seems to work too.
#31 Mis à jour par Matthieu Decorde il y a environ 4 ans
- % réalisé changé de 20 à 60
Linux and Windows tests passed
#32 Mis à jour par Sebastien Jacquot il y a presque 4 ans
- Statut changé de New à Closed
- % réalisé changé de 60 à 100
The solution is not fully satisfactory.
We need to match the R network configuration to the RCP network configuration.
See #1336.
#33 Mis à jour par Sebastien Jacquot il y a plus de 3 ans
Implementing the solution 1 means changing the configuration needs a full restart of R/Rserve.
Implementing the solution 3 means all the network commands in a R console won't be directly available for proxy users.
#34 Mis à jour par Sebastien Jacquot il y a plus de 3 ans
- Description mis à jour (diff)
- Statut changé de Closed à In Progress
- % réalisé changé de 100 à 80
#35 Mis à jour par Sebastien Jacquot il y a plus de 3 ans
At first we need to reduce the connection time out in R. See getOption("timeout"). Default seems to be 60 seconds in Linux.
#36 Mis à jour par Sebastien Jacquot il y a plus de 3 ans
To reduce R connection time out:
options(timeout = x);
with x in number of seconds.
TMP: for muli-target tests purpose:
print(Sys.getenv("http_proxy")); Sys.setenv(http_proxy="http://proxy-web.univ-fcomte.fr:3128"); print(Sys.getenv("http_proxy")); options(timeout = 1); print(getOption("timeout")); install.packages("textometry", dependencies=TRUE, repos="http://cran.rstudio.com/");
#37 Mis à jour par Sebastien Jacquot il y a plus de 3 ans
- Description mis à jour (diff)
#38 Mis à jour par Sebastien Jacquot il y a plus de 3 ans
Tests with last setup 0.7.7b2.2 Win and Linux:
print(Sys.getenv("http_proxy"));
returns well the proxy on Win and Linux.
install.packages("textometry", dependencies=TRUE, repos="http://cran.rstudio.com/");
freezes the UI on Win 7, Win XP and Linux.
The strange thing here is that on the Win XP machine, I have a firewall which asks me to authorize the connection to http://cran.rstudio.com/ when I type this line, but after the connection is accepted, the UI freezes until the R time out (about 30 minutes).
Log after the UI freeze is:
Warning: unable to access index for repository http://cran.rstudio.com/src/contrib
It seems there is some traffic on Linux, tcpdump log:
19:14:08.635729 IP ubuntu.local.6951 > 192.168.88.2.domain: 17681+ A? cran.rstudio.com. (34) 19:14:08.637274 IP 192.168.88.2.domain > ubuntu.local.6951: 17681 9/4/4 CNAME dqhc1zxg5b5jh.cloudfront.net., A 54.230.218.108, A 54.230.218.234, A 54.192.219.174, A 54.192.218.249, A 54.192.219.234, A 54.230.218.218, A 54.192.218.182, A 54.192.219.197 (402) 19:14:08.637421 IP ubuntu.local.34305 > server-54-230-218-108.mrs50.r.cloudfront.net.http: Flags [S], seq 3920735817, win 29200, options [mss 1460,sackOK,TS val 19121861 ecr 0,nop,wscale 7], length 0 19:14:09.612213 ARP, Request who-has 192.168.88.162 tell 192.168.88.2, length 46 19:14:09.635113 IP ubuntu.local.34305 > server-54-230-218-108.mrs50.r.cloudfront.net.http: Flags [S], seq 3920735817, win 29200, options [mss 1460,sackOK,TS val 19122111 ecr 0,nop,wscale 7], length 0 19:14:09.639198 IP ubuntu.local.54203 > server-54-230-218-234.mrs50.r.cloudfront.net.http: Flags [S], seq 1808043822, win 29200, options [mss 1460,sackOK,TS val 19122112 ecr 0,nop,wscale 7], length 0 ...
#39 Mis à jour par Sebastien Jacquot il y a plus de 3 ans
It seems:
install.packages("textometry", dependencies=TRUE, repos="http://cran.rstudio.com");
works on Linux when calling from R in a terminal but not in TXM therefore through RServe.
Useful option for debugging to change network communications log level?
options(internet.info = 0)
#40 Mis à jour par Sebastien Jacquot il y a plus de 3 ans
- Statut changé de In Progress à Feedback
- TBX tries to get the HTTP system proxy configuration from the RCP plugin org.eclipse.core.net native implementations and from the Java system properties if the first solution retrieved nothing
- R is launched without proxy setting
- after started, R time out is reduced from 60 seconds to 3 through RServe by calling 'options(timeout = 3);'
- after started, R HTTP proxy is configured through RServe by calling 'Sys.setenv(http_proxy="http://<user>:<password>@<host>:<port>")' using values of the retrieved system proxy settings
#41 Mis à jour par Sebastien Jacquot il y a plus de 3 ans
- Description mis à jour (diff)
#42 Mis à jour par Sebastien Jacquot il y a plus de 3 ans
Successfully validated with system proxy configurations on Win 7 64bit, Win XP 32bit, Ubuntu 14.04 (VM) and Mac OS X 10.9.4 (VM).
#43 Mis à jour par Sebastien Jacquot il y a presque 3 ans
- Statut changé de Feedback à Closed
- % réalisé changé de 80 à 100