You are not logged in.

Unanswered posts



Important! This site has been replaced. All content here is read-only. Please visit our brand-new community at https://community.talend.com/. We look forward to hearing from you there!



#1 2011-11-14 15:57:20

aiclara
Member
8 posts

aiclara said:

[resolved] POSTing a file to a website

Hi everyone!

I've got a task which I'd love to use TOS for but I can't for the life of me work out how to accomplish it.

There is a website (protected by apache basic auth) which requires me to POST a file (which happens to be a CSV). I can manage the first bit - getting the data, but I'm not sure about the second and third bits, namely forming this data into a CSV file and then using HTTP POST to send it to this other website.

For those that know Perl, here is a snippet of the code we have which is doing what I'd like TOS to do:

         
         my $ua = BrowserUA->new(
                 agent => "Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1",
         );

         my $req = POST $url,
                 "Content-Type" => 'multipart/form-data',
                 "Content" => [
                         submit => 1,
                         upfile => [ $output->{$cat}->{FILENAME} ],
                 ];

Has anyone got any hints that they can give me? I'm especially confused about the final bit of POSTing to the website as I can't see how to post a file (values aren't IN a file, I want to upload the actual file). I believe the content type above is making this happen, but I can't see how to do this in TOS.

Thanks in advance.

Offline

#2 2011-11-30 22:11:29

archenroot
Member
237 posts

archenroot said:

Re: [resolved] POSTing a file to a website

Hey,

first of all, what's your source data format?  When these data will be managed to CSV you could use tHttpRequest component which is able to do the POST file over HTTP protocol with custom headers if needed and then write response of the server to file, so you are able to check status of finished POST. This could be really easy, post some source data.

Ladislav

Last edited by archenroot (2011-11-30 22:13:42)


Emperor wants to control outer space Yoda wants to explore inner space that's the fundamental difference between good and bad sides of the Force

Offline

#3 2011-12-01 09:47:11

aiclara
Member
8 posts

aiclara said:

Re: [resolved] POSTing a file to a website

Hi,

Thanks for the reply.

For the proof of concept I'm doing, I'm short-cutting the generation of the file for now and I'm focussing only on POSTing the file to the receiving website. To do this, I've just hand-created a CSV file.

I have tried using tHttpRequest - even as the sole component in a job with everything hardcoded. In other words, I had for the Headers section:

"Content-Type"    "multipart/form-data"
"submit"   "1"
"upfile"  "C:\XXXXXX\YYYY.csv"

I also set:
Method: POST
And the appropriate authentication.

The job seems to run. It pauses after reporting "connected" (which I assume is my 104 line CSV file being uploaded) and some seconds later reports "disconnected" and exits with a code 0. However, the other system doesn't appear to have picked anything up.

That's currently where I am right now. If you have any advice for me on debugging this (or whether the way I've gone about this is wrong), then please let me know.

Thanks.

Offline

#4 2011-12-01 09:54:04

aiclara
Member
8 posts

aiclara said:

Re: [resolved] POSTing a file to a website

Oh, I forgot to mention that I've tried putting the server response back to a file and the file is 0 bytes....

Offline

#5 2011-12-01 10:08:25

aiclara
Member
8 posts

aiclara said:

Re: [resolved] POSTing a file to a website

A picture is worth 1000 words....

Offline

#6 2011-12-01 10:34:02

archenroot
Member
237 posts

archenroot said:

Re: [resolved] POSTing a file to a website

Well,

this component comes from my request in the past because I needed it also for my project, you can take a look what all was mentioned to implement at old bugtracker site http://www.talendforge.org/bugs/view.ph … 720#c63081. There were implemented only basic functions, but it's working just fine. The original souce code comes from Talend Exchange published component tFileHttpPost at http://www.talendforge.org/exchange/ind … &nav=2,1,1

I do not use custom headers in my scenario except one which indicates that I post text or XML (text/xml) content as you can see at attached picture.  Try to change your configuration in a way showed on picture, use "Post parameters from file", this is the attribute which point to file which schould be posted over http.

Ladislav

Last edited by archenroot (2011-12-01 10:36:28)


Emperor wants to control outer space Yoda wants to explore inner space that's the fundamental difference between good and bad sides of the Force

Offline

#7 2011-12-01 10:53:39

archenroot
Member
237 posts

archenroot said:

Re: [resolved] POSTing a file to a website

I also recommend use wireshark or other network capturing tool. Wireshark support function Follow TCP Stream, so once you find the packet related to your request, using this function wireshark show you whole request response, so you see what was sent and received.

Ladislav


Emperor wants to control outer space Yoda wants to explore inner space that's the fundamental difference between good and bad sides of the Force

Offline

#8 2011-12-01 10:57:26

archenroot
Member
237 posts

archenroot said:

Re: [resolved] POSTing a file to a website

If nothing from what proposed works, you could write your own subroutine in talend like this:

URL dest = "http://www.mysite.com "; 
URLConnection urlCon = dest.openConnection();

// prepare input and output 
urlCon.setDoInput(true); 
urlCon.setDoOutput(true);

// Disable caching 
urlCon.setUseCaches(false);

// Post output 
DataOutputStream out = 
new DataOutputStream(urlCon.getOutputStream()); 
out.writeBytes( file.toString ); 
out.flush(); 
out.close();

// return servlet response as a DataInputStream 
DataInputStream in = new DataInputStream(urlCon.getInputStream());

// and so on
....

There is also HTTP Client package provided by Apache Jakarta at http://hc.apache.org/downloads.cgi which make it easier to work with http.

Ladislav


Emperor wants to control outer space Yoda wants to explore inner space that's the fundamental difference between good and bad sides of the Force

Offline

#9 2011-12-01 11:05:52

aiclara
Member
8 posts

aiclara said:

Re: [resolved] POSTing a file to a website

Thanks for your info. I have tried re-doing my job to match more closely like yours but it's exactly the same situation.

I think I need to try WireShark to see what is going on as I get no feedback at all from TOS or the other website. I'd rather not have to write specific code for this, otherwise I will just stick with my Perl script smile

Offline

#10 2011-12-01 11:10:06

archenroot
Member
237 posts

archenroot said:

Re: [resolved] POSTing a file to a website

Wireshark will show the pure traffic, but you could also try to debug Talend session. When starting job from Talend you could use Debug trace or pure debug session to see what's go'in on. Let me know the results...

Ladislav


Emperor wants to control outer space Yoda wants to explore inner space that's the fundamental difference between good and bad sides of the Force

Offline

#11 2011-12-05 10:01:14

aiclara
Member
8 posts

aiclara said:

Re: [resolved] POSTing a file to a website

I've solved this!!

The key was not to use tHttpRequest at all. The correct component to use is tFileFetch. Despite it's name, it can send a file too. Worked straight away, very simple.

Offline

#12 2011-12-05 10:08:50

archenroot
Member
237 posts

archenroot said:

Re: [resolved] POSTing a file to a website

Nice :-) But still I don't know why the tHttpRequest didn't work for you.


Emperor wants to control outer space Yoda wants to explore inner space that's the fundamental difference between good and bad sides of the Force

Offline

#13 2011-12-05 10:14:48

aiclara
Member
8 posts

aiclara said:

Re: [resolved] POSTing a file to a website

Yep, that is a mystery, and if I needed to POST the file as well as other things, I may well have been stuck. I might look into it one day if I get the time. For now though, I can get on with my project smile

Offline

#14 2013-02-23 00:02:15

jijli
Member
104 posts

jijli said:

Re: [resolved] POSTing a file to a website

Hi aiclara,
I need to to do the same thing as you did. Can you show me how to use tFileFetch instead of tHttpRequest?

Thanks.

Offline

#15 2016-06-29 01:24:30

smurthy1996
Member
2 posts

smurthy1996 said:

Re: [resolved] POSTing a file to a website

You guysssssss, come on since when can you make a http request to a https URL. Haha lol your missing a whole  layer of encryption. If no one has figured it out after a difference of more than 3 years! Cheers.

Offline

Board footer

Talend Contributor Agreement - Talend Website Privacy Policy