Ar.Drone 2.0 video data without the SDK?

Added by Jeremy Graham 672 days ago

Has anyone figured out how to get the video stream data from the Ar.Drone 2.0 without using the SDK? I am trying to do this and I can not even seem to get any response from the drone when I send a packet to port 5555; no data comes in from the drone.


Replies

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 672 days ago

Also no Navigation data feedback? for me, i am also writing my own programming without library. but, i can only control ar drone 2, no nav-data and video feedback. are u using udp? do u have any idea to improve the udp?

RE: Ar.Drone 2.0 video data without the SDK? - Added by Christian Taddey 672 days ago

What data are you sending to 5555? Are you sure you're sending the 'right' initialization data?

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 672 days ago

Jiahui: I haven't tried to get navigational data yet. I'm sorry I can not comment on that as of yet. What exactly do you mean by "improve the UDP"? I am not a network expert by any means whatsoever but maybe I can help.

Jiahui and Christian: It appears that using TCP rather than UDP to port 5555 has been successful for me. I don't know what exactly I am getting and I don't know what to do with it, but I am in fact getting a stream of some kind of data from that port. Assuming it is video data, can either of you point me to some information that might explain what to do with it?

Thanks for your replies! :)

RE: Ar.Drone 2.0 video data without the SDK? - Added by Will Ross 672 days ago

It's standard-ish H.264. It can be decoded using a recent ffmpeg (older ones have some issues with it and only decode the key frames). I don't know if any other decoders will process it, just that a recent ffmpeg does.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Will Ross 672 days ago

Also, in my experience the drone sends out the video stream without any prompting.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 672 days ago

Interesting. Thanks for the reply, Will.

I am a total novice when it comes to video (including compression/decompression/encoding/decoding). Are there any accessible and easy to understand sources of info that you might recommend I look at so that I can figure out how it all works?

Also, if ffmpeg can decode it, does that mean that I only have to decode the video stream that I get from the drone with ffmpeg and I can get video frames as images?

RE: Ar.Drone 2.0 video data without the SDK? - Added by Dardo Kleiner 671 days ago

It is indeed H.264 over TCP - rather odd I think for a real-time video stream.

Will, its actually the modified ffmpeg in gstreamer's kit that seems to have trouble - I did test as old as 0.7.2 ffmpeg release and it worked fine.

Jeremy - here's a good idea: take http://code.google.com/p/pyffmpeg and contribute fixes to https://github.com/venthur/python-ardrone.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 671 days ago

Thanks for that confirmation, Dardo. Would you happen to have any code samples that are in C/C++ or Java? I do not know Python I am afraid. I'll see what info I can get out of the links you posted though.

So in trying to figure out how to decode H.264 from a network stream, I came across this link: http://www.mail-archive.com/libav-user@mplayerhq.hu/msg04939.html
It mentions the following thing: "To decode H.264 stream you need to have SPS and PPS NAL units also."
Can anyone tell me where I can find out what SPS and PPS NAL units are, how to get that info from a stream, and how to use it? I have found some info on the first item but it was not really clear to me what it was talking about. I have not found anything yet on the last two items.

Thanks everyone for the responses in this thread!

RE: Ar.Drone 2.0 video data without the SDK? - Added by Dardo Kleiner 670 days ago

Are you trying to write your own H.264 decoder? This is not the right forum for discussing that (and IMHO you'd be going down a very difficult path, especially as "a total novice when it comes to video"). If you're just trying to access the video stream from the 2.0 AR.Drone, I highly recommend leveraging an existing library like ffmpeg (search for "ffmpeg tutorial"). There are Java bindings also (http://fmj-sf.net/ffmpeg-java/getting_started.php), easy to find with a quick search.

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 670 days ago

i managed to convert android free flight APK file into java. But any one have idea how to do video part by using udp?
https://www.dropbox.com/home/src

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 670 days ago

any one know what have been changed of AR Drone 2.0?
i mean for those ports and commands?

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 670 days ago

jeremy, does it work when u use that python way to decode?

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 670 days ago

Dardo: No sir, I am not trying to write a decoder. I just want to be able to grab the frames of video data from the drone (as images). I'll try that search that you recommended and look at the Java code though I have essentially searched for info about this and came across links like these: http://dranger.com/ffmpeg/tutorial01.html http://libav.org/doxygen/master/libavcodec_2api-example_8c-example.html http://www.inb.uni-luebeck.de/~boehme/using_libavcodec.html. Actually, I tried to do the job using code that closely follows theirs and I get output like this as my program is receiving the stream:

[h264 @ 0xb0c00480] out of range intra chroma pred mode at 15 1
[h264 @ 0xb0c00480] error while decoding MB 15 1
[h264 @ 0xb0c00480] concealing 914 DC, 914 AC, 914 MV errors
[h264 @ 0xb0c00480] negative number of zero coeffs at 27 17
[h264 @ 0xb0c00480] error while decoding MB 27 17
[h264 @ 0xb0c00480] concealing 262 DC, 262 AC, 262 MV errors
[h264 @ 0xb0c00480] concealing 354 DC, 354 AC, 354 MV errors
[h264 @ 0xb0c00480] no frame!
Error while decoding frame data
[h264 @ 0xb0c00480] concealing 336 DC, 336 AC, 336 MV errors
[h264 @ 0xb0c00480] no frame!
Error while decoding frame data
[h264 @ 0xb0c00480] concealing 407 DC, 407 AC, 407 MV errors
[h264 @ 0xb0c00480] no frame!
Error while decoding frame data
[h264 @ 0xb0c00480] concealing 656 DC, 656 AC, 656 MV errors
[h264 @ 0xb0c00480] no frame!
Error while decoding frame data
[h264 @ 0xb0c00480] concealing 564 DC, 564 AC, 564 MV errors
[h264 @ 0xb0c00480] no frame!
Error while decoding frame data
[h264 @ 0xb0c00480] concealing 384 DC, 384 AC, 384 MV errors
[h264 @ 0xb0c00480] no frame!
Error while decoding frame data
[h264 @ 0xb0c00480] concealing 458 DC, 458 AC, 458 MV errors
[h264 @ 0xb0c00480] no frame!
Error while decoding frame data
[h264 @ 0xb0c00480] concealing 380 DC, 380 AC, 380 MV errors
[h264 @ 0xb0c00480] no frame!
Error while decoding frame data

So I must be doing something wrong; problem is, I am not sure where to go from here. I've encountered statements like, "To decode H.264 stream you need to have SPS and PPS NAL units also" a few times. Is that the problem? Do I need to get that information somehow and use it somewhere in order to get ffmpeg to decode the stream?

By the way, you mentioned that you've had success using ffmpeg. Were you successful in: using ffmpeg libraries in a program to decode frames as they come, running the ffmpeg executable with a command like "ffmpeg -i http://192.168.1.1:5555" while the drone is streaming, or did you run the ffmpeg executable offline on data received from the drone? If you wish I could post the part of my code that is attempting to decode the stream but I do not know if you were wanting to go that far.

Thanks for your input!

Jiahui: I haven't tried the Python code yet, unfortunately.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Dardo Kleiner 669 days ago

I was successful in decoding frames as they come, using "ffplay http://192.168.1.1:5555". Then I was successful in decoding the live stream within a gstreamer-based (http://gstreamer.freedesktop.org/) application once I replaced its internal ffmpeg with a stock release (0.7.2 in this case was closest to gstreamer's own). A stream is a stream is a stream, if you're sufficiently feeding the "black box" that is the decoder it should just give you the frames when they're ready. In our sniffs I've seen nothing that indicates additional video information being conveyed outside of the TCP/5555 connection.

HTH!

- Dardo

RE: Ar.Drone 2.0 video data without the SDK? - Added by tom sparks 669 days ago

Dardo Kleiner wrote:

I was successful in decoding frames as they come, using "ffplay http://192.168.1.1:5555". Then I was successful in decoding the live stream within a gstreamer-based (http://gstreamer.freedesktop.org/) application once I replaced its internal ffmpeg with a stock release (0.7.2 in this case was closest to gstreamer's own). A stream is a stream is a stream, if you're sufficiently feeding the "black box" that is the decoder it should just give you the frames when they're ready. In our sniffs I've seen nothing that indicates additional video information being conveyed outside of the TCP/5555 connection.

HTH!

- Dardo

ffplay works :)
until the SDK for ar.drone 2.0 comes out this well be my only video option :(

- tom

RE: Ar.Drone 2.0 video data without the SDK? - Added by Christian Taddey 669 days ago

Good to know.
Also the VLC player might work. I have used it to connect to h.264 streams in the past.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Dardo Kleiner 669 days ago

tom - I bet when the 2.0 SDK is released it will use ffmpeg under the hood for Linux, DirectShow for Windows, and platform SDK video decoders for Android & iPhone. The move to a "standard" video format is surely so Parrot doesn't have to write/maintain their own decoder. Particularly since (I believe) they're using the stock hardware DSP TMS320DMC64x for H.264 encoding on the device.

Christian - I think Will tried VLC with no success, but I don't know what was wrong with it (whether it was poor decoding or none at all). I also think VLC is harder to integrate into other applications as a library, but I've not worked extensively with it to say for sure.

RE: Ar.Drone 2.0 video data without the SDK? - Added by leor leor 668 days ago

There is a link to a github of sdk1.8 here:
https://projects.ardrone.org/boards/1/topics/show/3947#message-4316
https://github.com/johnboiles/ARDroneSDK

Would be cool to add an option to use tcp and a ffmpeg lib h.264 decoder there, like it would make parrot and sdk 2.0 almost a non-issue. (I think there is some nav stuff, like the new altitude sensor).

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jörg Sawatzki 668 days ago

Hey guys,

is anybody using Ubuntu 12.04? I'm trying to get a video stream from the drone with ffmpeg, but I unfortunately don't have any success, I see a still image from the camera or a gray screen. Any ideas/suggestions?

Do I need to send any control commands to the drone before the stream works? Does it need to be filtered somehow?

Thanks in advance,

Jörg

ffplay http://192.168.1.1:5555
avplay version 0.8.3-4:0.8.3-0ubuntu0.12.04.1, Copyright (c) 2003-2012 the Libav developers
built on Jun 12 2012 16:52:09 with gcc 4.6.3
[h264 @ 0x7f8934000e00] concealing 920 DC, 920 AC, 920 MV errors
Last message repeated 5 times
[h264 @ 0x7f89340008c0] max_analyze_duration reached
[h264 @ 0x7f89340008c0] Estimating duration from bitrate, this may be inaccurate
Input #0, h264, from 'http://192.168.1.1:5555':
Duration: N/A, bitrate: N/A
Stream #0.0: Video: h264 (Baseline), yuv420p, 640x360, 25 fps, 25 tbr, 1200k tbn, 2400k tbc
[h264 @ 0x7f8934006e60] concealing 920 DC, 920 AC, 920 MV errorsf=0/0
[h264 @ 0x7f8934007940] concealing 920 DC, 920 AC, 920 MV errors
[h264 @ 0x7f8934007f40] concealing 920 DC, 920 AC, 920 MV errors
[h264 @ 0x7f8934006e60] concealing 920 DC, 920 AC, 920 MV errorsf=0/0
[h264 @ 0x7f8934007940] concealing 920 DC, 920 AC, 920 MV errors

RE: Ar.Drone 2.0 video data without the SDK? - Added by tom sparks 668 days ago

did you compile ffmpeg using these steps http://ffmpeg.org/trac/ffmpeg/wiki/UbuntuCompilationGuide
and do this
sudo cp -rf /usr/include/libavformat /usr/local/include 

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jörg Sawatzki 668 days ago

well, I didn't compile ffmpeg at all....just took the binaries from the distribution....:)

RE: Ar.Drone 2.0 video data without the SDK? - Added by tom sparks 668 days ago

Dardo Kleiner wrote:

tom - I bet when the 2.0 SDK is released it will use ffmpeg under the hood for Linux, DirectShow for Windows, and platform SDK video decoders for Android & iPhone. The move to a "standard" video format is surely so Parrot doesn't have to write/maintain their own decoder. Particularly since (I believe) they're using the stock hardware DSP TMS320DMC64x for H.264 encoding on the device.

that may be so, but h.264 has patents ( http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Patent_licensing ), the P264 codec may just be different enough to escape the patents but still be compatible with h.264 decoders

-tom

RE: Ar.Drone 2.0 video data without the SDK? - Added by tom sparks 668 days ago

Jörg Sawatzki wrote:

well, I didn't compile ffmpeg at all....just took the binaries from the distribution....:)

ubuntu ffmpeg binaries are allways out of date

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jörg Sawatzki 668 days ago

hehe....you are absolutely right ;)
I just compiled the latest git checkout and everything works....now the fun can begin with python and opencv ;)
thanks,
jörg

RE: Ar.Drone 2.0 video data without the SDK? - Added by Dardo Kleiner 668 days ago

tom sparks wrote:

that may be so, but h.264 has patents ( http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Patent_licensing ), the P264 codec may just be different enough to escape the patents but still be compatible with h.264 decoders

I think they exactly avoid this problem by switching to TI's hardware DSP encoder for which I'm sure TI has paid licensing. And on the decoder side I'm saying they're going to leave it to the platform implementations instead of even messing with the hope of their own being different enough. Many Android platforms use the same TI tech, and you can be sure that Apple & Microsoft have their ducks in a row wrt licensing for their implementations. IANAL, of course...

RE: Ar.Drone 2.0 video data without the SDK? - Added by tom sparks 668 days ago

Dardo Kleiner wrote:

tom sparks wrote:

that may be so, but h.264 has patents ( http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Patent_licensing ), the P264 codec may just be different enough to escape the patents but still be compatible with h.264 decoders

I think they exactly avoid this problem by switching to TI's hardware DSP encoder for which I'm sure TI has paid licensing. And on the decoder side I'm saying they're going to leave it to the platform implementations instead of even messing with the hope of their own being different enough. Many Android platforms use the same TI tech, and you can be sure that Apple & Microsoft have their ducks in a row wrt licensing for their implementations. IANAL, of course...

good point, that just adds more headache for linux/floss developers

-tom

RE: Ar.Drone 2.0 video data without the SDK? - Added by Dardo Kleiner 668 days ago

tom sparks wrote:

good point, that just adds more headache for linux/floss developers

H.264 is bad for FOSS, IMHO. It would've been interesting had Parrot embraced a patent unencumbered format like VP8. But the state of things is clearly that H.264 is winning that battle. Particularly in hardware implementations, and leveraging those just makes commercial sense.

But for now, we do have FFmpeg, and I think it's actually a good thing that we don't have to incorporate Parrot's custom P.264 in a 2.0 AR.Drone client. As long as you're not trying to commercialize a product around it (http://ffmpeg.org/legal.html), you're likely fine!

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 667 days ago

hi, is anyone writing their own programming to control Ar Drone2 by TCP, I got something problem when i setting the command TCP port 5556, i can't connect to the drone. here is my coding:
bool TheirPortTCP::init( unsigned short broadcastPortNumber, int b1, int b2, int b3, int b4 ) {
this->broadcastPortNumber = broadcastPortNumber;
this->b1 = b1;
this->b2 = b2;
this->b3 = b3;
this->b4 = b4; }

// Open Windows connection
if( WSAStartup( 0x0101, &w ) != 0 ) return( false );
ShowMessage( "WSAStartup OK!" );
// Set family and port
broadcast.sin_family = AF_INET;
broadcast.sin_port = htons( broadcastPortNumber );
// Assign the address  [B1,B2,B3,B4]
broadcast.sin_addr.S_un.S_un_b.s_b1 = ( unsigned char )b1;
broadcast.sin_addr.S_un.S_un_b.s_b2 = ( unsigned char )b2;
broadcast.sin_addr.S_un.S_un_b.s_b3 = ( unsigned char )b3;
broadcast.sin_addr.S_un.S_un_b.s_b4 = ( unsigned char )b4;
broadcastLength = ( int )sizeof( struct sockaddr_in );
// CREATE socket
sd = socket( AF_INET, SOCK_STREAM, IPPROTO_TCP );
if( sd INVALID_SOCKET ) return( false );
ShowMessage( "Socket valid!" );
// try connecting
if( connect( sd, ( SOCKADDR *)&broadcast, sizeof( broadcast)) SOCKET_ERROR ) {
ShowMessage( "not connected!" );
return false;//couldn't connect
} else {
ShowMessage( "connected!" );
return true;
}

is anyone know what is the problem? thanks. the address i use is 192.168.1.1 and port number is 5556.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jörg Sawatzki 667 days ago

Hi Jiahui,

as far as I know, the drone receives the AT commands through UDP, not TCP as you try in your code.

Jörg

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 667 days ago

Dang, I am not getting video by connecting to the drone via Wifi and then typing "ffplay http://192.168.1.1:5555" in a terminal window. I did follow those exact instructions posted by that link above (http://ffmpeg.org/trac/ffmpeg/wiki/UbuntuCompilationGuide) to get the latest ffmpeg and I even executed that copy command that was mentioned. Does anyone have any ideas as to what could be the problem?

I let ffplay run for nearly 22 seconds and all that it displayed was a solid black window. The output on terminal during that time was the following:

ffplay version git-2012-06-20-b2e2287 Copyright (c) 2003-2012 the FFmpeg developers
built on Jun 20 2012 17:25:25 with gcc 4.6.3
configuration: --enable-gpl --enable-libfaac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree --enable-version3 --enable-x11grab
libavutil 51. 59.100 / 51. 59.100
libavcodec 54. 27.100 / 54. 27.100
libavformat 54. 9.100 / 54. 9.100
libavdevice 54. 0.100 / 54. 0.100
libavfilter 2. 82.100 / 2. 82.100
libswscale 2. 1.100 / 2. 1.100
libswresample 0. 15.100 / 0. 15.100
libpostproc 52. 0.100 / 52. 0.100
[h264 @ 0xb1500480] max_analyze_duration 5000000 reached at 5000000
[h264 @ 0xb1500480] Estimating duration from bitrate, this may be inaccurate
Input #0, h264, from 'http://192.168.1.1:5555':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: h264 (Baseline), yuv420p, 640x360, 25 fps, 25 tbr, 1200k tbn, 50 tbc
[ffplay_buffer @ 0xb2400700] w:640 h:360 pixfmt:yuv420p tb:1/1200000 fr:0/1 sar:0/1 sws_param:
[h264 @ 0xb5bc7aa0] left block unavailable for requested intra4x4 mode -1 at 0 17
[h264 @ 0xb5bc7aa0] error while decoding MB 0 17
[h264 @ 0xb5bc7aa0] concealing 289 DC, 289 AC, 289 MV errors
[h264 @ 0xb50af740] out of range intra chroma pred mode at 7 2
[h264 @ 0xb50af740] error while decoding MB 7 2
[h264 @ 0xb50af740] concealing 882 DC, 882 AC, 882 MV errors
[h264 @ 0xb1501980] FMO not supportedB vq= 0KB sq= 0B f=0/0
[h264 @ 0xb1501980] reference overflow (pps)
[h264 @ 0xb5bc7aa0] FMO not supported
[h264 @ 0xb5bc7aa0] reference overflow (pps)
[h264 @ 0xb1501980] sps_id (32) out of range 0KB sq= 0B f=0/0
[h264 @ 0xb150a780] sps_id (32) out of range
Last message repeated 1 times 0KB vq= 0KB sq= 0B f=0/0
[h264 @ 0xb1501980] FMO not supported
[h264 @ 0xb1501980] reference overflow (pps)
[h264 @ 0xb50af740] FMO not supported
[h264 @ 0xb50af740] reference overflow (pps)
[h264 @ 0xb1501980] sps_id (32) out of range 0KB sq= 0B f=0/0
[h264 @ 0xb5b9d840] sps_id (32) out of range
21.97 A-V: 0.000 fd= 0 aq= 0KB vq= 0KB sq= 0B f=0/0

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jörg Sawatzki 667 days ago

Hi Jeremy,

this is what my ffplay says:

ffplay version N-41863-gf8fe4d2 Copyright (c) 2003-2012 the FFmpeg developers
built on Jun 24 2012 02:10:17 with gcc 4.6.3
configuration: --enable-gpl --enable-libfaac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree --enable-version3 --enable-x11grab
libavutil 51. 61.100 / 51. 61.100
libavcodec 54. 29.100 / 54. 29.100
libavformat 54. 11.100 / 54. 11.100
libavdevice 54. 0.100 / 54. 0.100
libavfilter 2. 82.100 / 2. 82.100
libswscale 2. 1.100 / 2. 1.100
libswresample 0. 15.100 / 0. 15.100
libpostproc 52. 0.100 / 52. 0.100

I followed the Ubuntu compilation instructions you mentioned as well and it worked out of the box.
Have you tried to build it from a newer git checkout?

Have you got the latest AR.FreeFlight und updated the drone's firmware with it?

Jörg

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 667 days ago

I just gave it another try after first updating ffmpeg (again using the update instructions at that link) and updating the firmware via the AR.FreeFlight 2.0 beta app on a Samsung Galaxy S phone. The behavior is still the same. What could I possibly be doing differently from those of you who have been successful?

Should I send the drone a packet first? I think someone mentioned already that the TCP stream is automatic now; and it sounds like several of you did not have to do that. This is so frustrating. :(

RE: Ar.Drone 2.0 video data without the SDK? - Added by tom sparks 667 days ago

mine drone software is software has NOT be upgraded
tom

ps: make sure there is one copy of libavformat on the system by using
sudo cp -rf /usr/include/libavformat /usr/local/include 

RE: Ar.Drone 2.0 video data without the SDK? - Added by tom sparks 667 days ago

tom sparks wrote:
my drone software/firmware has NOT be upgraded (factory firmware)

tom

ps: make sure there is one copy of libavformat on the system by using [...]

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 667 days ago

So I guess the drone is good to go out of the box. I suppose I can eliminate that from the list of possible problems.

I actually did copy the libavformat folder to /usr/local/include but that did not seem to do anything for me. I still get errors from ffplay and no video.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jörg Sawatzki 666 days ago

Hmm, I actually did not copy anything to /usr/local/include.
I basically just deinstalled the ffmpeg stuff from the distribution (as described in the compilation guide as well) and generated new packages with the checkinstall commands (also taken from the compilation guide)...

We could easily check if your drone is OK if you do something like this:
nc 192.168.1.1 5555 > test.h264
(and press CTRL+C after a few seconds of video recording)

Then attach the test.h264 file here and we can check if it can be played back with our ffmpeg versions....

Jörg

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 666 days ago

Great idea! I'll do that tomorrow when I have access to the drone.

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 666 days ago

hi,Jeremy, how did you write your TCP part for port5555? my sending and receiving part all shows "-1" after i changed my udp to tcp. do you have any ideas what did i do wrong?

bool OurPortTCP::init( unsigned short broadcastPortNumber, int b1, int b2, int b3, int b4 ) {
this->b1 = b1;
this->b2 = b2;
this->b3 = b3;
this->b4 = b4;
this->broadcastPortNumber = broadcastPortNumber;

// open Windows connection
if( WSAStartup( 0x0101, &w ) != 0 ) return( false );
// set family and port
broadcast.sin_family = AF_INET;
broadcast.sin_port = htons( broadcastPortNumber );
// assign the server address
broadcast.sin_addr.S_un.S_un_b.s_b1 = ( unsigned char )b1;
broadcast.sin_addr.S_un.S_un_b.s_b2 = ( unsigned char )b2;
broadcast.sin_addr.S_un.S_un_b.s_b3 = ( unsigned char )b3;
broadcast.sin_addr.S_un.S_un_b.s_b4 = ( unsigned char )b4;
broadcastLength = ( int )sizeof( struct sockaddr_in );
//Create socket
sd = socket( AF_INET, SOCK_STREAM, IPPROTO_TCP );
if( sd INVALID_SOCKET ) return( false );
//ShowMessage( "Socket valid!" );
// bind address to socket
if( bind( sd, ( struct sockaddr * )&broadcast, sizeof( struct sockaddr_in ) ) SOCKET_ERROR ) {
ShowMessage( " not connected " );
return( false );
}else {
ShowMessage ( " connected ");
return ( true );
}

}
int OurPortTCP::send( unsigned char *image, int byte ) {
int y;
y = sendto( sd, ( char * )image, byte,
0, ( struct sockaddr * )&broadcast, broadcastLength );
ShowMessage( y );
return ( y );
}
int OurPortTCP::receive( unsigned char *image, int byte ) {
int x;
x = recvfrom( sd, ( char * )image, byte, 0, ( struct sockaddr * )&broadcast, &broadcastLength );
// return( recvfrom( sd, ( char * )image, byte, 0, ( struct sockaddr * )&broadcast, 0 ) );
//ShowMessage ( x );
return ( x );
}

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 666 days ago

Jiahui, I am just taking sort of a guess here, but is it possible that you need to call some sort of connect function (or method) to establish the TCP connection between your computer and the drone?

RE: Ar.Drone 2.0 video data without the SDK? - Added by leor leor 666 days ago

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 666 days ago

Okay, attached to this post is a stream of video data received from my drone using the nc (netcat) command. Can anyone else decode this using their versions of ffmpeg?

videotest.h264 (1.8 MB)

RE: Ar.Drone 2.0 video data without the SDK? - Added by tom sparks 666 days ago

playable by ffmpeg
ffmpeg version git-2012-06-23-e9d5c05 Copyright (c) 2000-2012 the FFmpeg developers
  built on Jun 23 2012 12:04:42 with gcc 4.6.3
  configuration: --enable-gpl --enable-libfaac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree --enable-version3 --enable-x11grab
  libavutil      51. 61.100 / 51. 61.100
  libavcodec     54. 29.100 / 54. 29.100
  libavformat    54. 11.100 / 54. 11.100
  libavdevice    54.  0.100 / 54.  0.100
  libavfilter     2. 82.100 /  2. 82.100
  libswscale      2.  1.100 /  2.  1.100
  libswresample   0. 15.100 /  0. 15.100
  libpostproc    52.  0.100 / 52.  0.100
[h264 @ 0xb129660] sps_id (32) out of range
[h264 @ 0xb1233e0] max_analyze_duration 5000000 reached at 5000000
[h264 @ 0xb1233e0] Estimating duration from bitrate, this may be inaccurate
Input #0, h264, from 'videotest.mp4':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: h264 (Baseline), yuv420p, 640x360, 25 fps, 25 tbr, 1200k tbn, 50 tbc

also playable by ffplay
ffplay version git-2012-06-23-e9d5c05 Copyright (c) 2003-2012 the FFmpeg developers
  built on Jun 23 2012 12:04:42 with gcc 4.6.3
  configuration: --enable-gpl --enable-libfaac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree --enable-version3 --enable-x11grab
  libavutil      51. 61.100 / 51. 61.100
  libavcodec     54. 29.100 / 54. 29.100
  libavformat    54. 11.100 / 54. 11.100
  libavdevice    54.  0.100 / 54.  0.100
  libavfilter     2. 82.100 /  2. 82.100
  libswscale      2.  1.100 /  2.  1.100
  libswresample   0. 15.100 /  0. 15.100
  libpostproc    52.  0.100 / 52.  0.100
[h264 @ 0xb2301980] sps_id (32) out of range
[h264 @ 0xb2300480] max_analyze_duration 5000000 reached at 5000000
[h264 @ 0xb2300480] Estimating duration from bitrate, this may be inaccurate
Input #0, h264, from 'videotest.mp4':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: h264 (Baseline), yuv420p, 640x360, 25 fps, 25 tbr, 1200k tbn, 50 tbc
[ffplay_buffer @ 0xb3e00700] w:640 h:360 pixfmt:yuv420p tb:1/1200000 fr:0/1 sar:0/1 sws_param:

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 665 days ago

hi, Jeremy, thank you.now i can receive data .

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 665 days ago

You're welcome, Jiahui. Glad it worked for you.

So you can see my video file just fine, Tom? What in the world could possibly be going on such that your version of ffmpeg/ffplay can manage the file yet mine can not?

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 665 days ago

now i can receive some data from port 5555, and i downloaded the ffmpeg 0.8. But i am using turbo c ++, how to include the ffmpeg? just copy all files into my program, and then call function? thanks

RE: Ar.Drone 2.0 video data without the SDK? - Added by tom sparks 665 days ago

Jeremy Graham wrote:

You're welcome, Jiahui. Glad it worked for you.

So you can see my video file just fine, Tom?

yes

What in the world could possibly be going on such that your version of ffmpeg/ffplay can manage the file yet mine can not?

run ffmpeg in the terminal and post it output (you dont need your drone)
here is mine
ffmpeg version git-2012-06-23-e9d5c05 Copyright (c) 2000-2012 the FFmpeg developers
  built on Jun 23 2012 12:04:42 with gcc 4.6.3
  configuration: --enable-gpl --enable-libfaac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree --enable-version3 --enable-x11grab
  libavutil      51. 61.100 / 51. 61.100
  libavcodec     54. 29.100 / 54. 29.100
  libavformat    54. 11.100 / 54. 11.100
  libavdevice    54.  0.100 / 54.  0.100
  libavfilter     2. 82.100 /  2. 82.100
  libswscale      2.  1.100 /  2.  1.100
  libswresample   0. 15.100 /  0. 15.100
  libpostproc    52.  0.100 / 52.  0.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

tom

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 665 days ago

I typed in "ffmpeg -i videotest.h264" and this is what was output along with no video:

ffmpeg version git-2012-06-26-37c3864 Copyright (c) 2000-2012 the FFmpeg developers
built on Jun 26 2012 12:13:31 with gcc 4.6.3
configuration: --enable-gpl --enable-libfaac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree --enable-version3 --enable-x11grab
libavutil 51. 63.100 / 51. 63.100
libavcodec 54. 29.101 / 54. 29.101
libavformat 54. 11.100 / 54. 11.100
libavdevice 54. 0.100 / 54. 0.100
libavfilter 3. 0.100 / 3. 0.100
libswscale 2. 1.100 / 2. 1.100
libswresample 0. 15.100 / 0. 15.100
libpostproc 52. 0.100 / 52. 0.100
[h264 @ 0x9b633e0] Format h264 detected only with low score of 1, misdetection possible!
[h264 @ 0x9b633e0] Could not find codec parameters (Video: h264)
[h264 @ 0x9b633e0] Estimating duration from bitrate, this may be inaccurate
videotest.h264: could not find codec parameters

RE: Ar.Drone 2.0 video data without the SDK? - Added by tom sparks 665 days ago

something is wrong with your version of ffmpeg
attached is my ffmpeg executable
save ffmpeg into the folder where your videotest.h264 file is
run this command in you terminal inside the folder where videotest.h264 file is
./ffmpeg -i "videotest.h264" 

ffmpeg (11.3 MB)

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 664 days ago

Actually that didn't work either. I don't know what it is. I get video with the FreeFlight 2.0 beta app on my Galaxy S phone and I get it when I use that ardrone_navigation program that I compiled with the newly released SDK 2.0, but I can't see it when I try to execute ffmpeg or ffplay in a terminal window.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 664 days ago

I noticed that pages 59-60 of the new developers guide mentions that, "For network transmission, video frames are send with custom headers, which contains many
informations about the frame."

This doesn't explain why all of you are not having issues but I wonder if this is the cause of my problems. Could it be that the ffmpeg libraries can not decode these packets due to the presence of Parrot's custom header? Do I need to strip that off somehow before I send the packets to the decoder?

RE: Ar.Drone 2.0 video data without the SDK? - Added by tom sparks 664 days ago

Jeremy Graham wrote:

Actually that didn't work either. I don't know what it is. I get video with the FreeFlight 2.0 beta app on my Galaxy S phone and I get it when I use that ardrone_navigation program that I compiled with the newly released SDK 2.0, but I can't see it when I try to execute ffmpeg or ffplay in a terminal window.

you'll have uninstall ffmpeg from ubuntu via apt-get
and recompile it with step give earlier

RE: Ar.Drone 2.0 video data without the SDK? - Added by tom sparks 664 days ago

Jeremy Graham wrote:

I noticed that pages 59-60 of the new developers guide mentions that, "For network transmission, video frames are send with custom headers, which contains many informations about the frame."

This doesn't explain why all of you are not having issues but I wonder if this is the cause of my problems. Could it be that the ffmpeg libraries can not decode these packets due to the presence of Parrot's custom header? Do I need to strip that off somehow before I send the packets to the decoder?

no ffmpeg/ffplay whould noticed if the frame were damaged/missing/incorrect and would not decode those frames and wait for clean/decode-able frames

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 663 days ago

has anyone found which file inside the SDK 2.0 is writing about receiving data form tcp? and where do they store those data? Thanks

i read through those video files, basically, they have vp_api_io_data_t *in and vp_api_io_data_t *out, looks like they put data from tcp into *in, but i couldn't find out where they do this, and if the data from tcp is not under*in, where are they?

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 660 days ago

I sent this message to an ffmpeg developement mailing list last week and, so far, I have silence. Maybe someone here can say a thing or two?

"Hello, everyone. I am sure that I am asking very basic questions that you all have hashed over time and time again. However, try as I might, I can't seem to find a source of info out there which clearly and specifically answers my questions. Perhaps someone out out there can help me to understand the following.

What do general messages about concealing errors from avcodec_decode_video2 calls mean? Should I be concerned about them? In my program, the function returns values that are greater than 0 yet it also prints out this message every time: [h264 @ 0xb0b00480] concealing 920 DC, 920 AC, 920 MV errors.

What does it mean when sws_scale outputs an error message like: [swscaler @ 0xb0b505a0] bad dst image pointers? Is there something very particular about the way that it expects to receive the parameter for the destination?"

RE: Ar.Drone 2.0 video data without the SDK? - Added by tom sparks 660 days ago

Jeremy Graham wrote:

I sent this message to an ffmpeg developement mailing list last week and, so far, I have silence. Maybe someone here can say a thing or two?

"Hello, everyone. I am sure that I am asking very basic questions that you all have hashed over time and time again. However, try as I might, I can't seem to find a source of info out there which clearly and specifically answers my questions. Perhaps someone out out there can help me to understand the following.

What do general messages about concealing errors from avcodec_decode_video2 calls mean? Should I be concerned about them? In my program, the function returns values that are greater than 0 yet it also prints out this message every time: [h264 @ 0xb0b00480] concealing 920 DC, 920 AC, 920 MV errors.

you don't need to worry about them, h264 decoder is say the stream is damaged and is trying to hide/conceal the damage
you only need to worry if the I-frames/keyframes are damaged or get a high-level of macroblocking
see these images for examples
http://i.umumble.com/img/topic-1324999650.png
http://upload.wikimedia.org/wikipedia/commons/b/b1/The_macroblocking_effect_%28JPEG%29.png

tom

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 660 days ago

Hey, Tom. Thanks for the response!

So is the stream actually damaged, out of curiosity, or is the decoder just saying that?

Also, would you happen to know about the second thing I ask about? That is the one that I am really trying to figure out at the moment. Do you know what is going on when sws-scale tells you that you have "bad dst image pointers"?

RE: Ar.Drone 2.0 video data without the SDK? - Added by tom sparks 659 days ago

Jeremy Graham wrote:

Hey, Tom. Thanks for the response!

So is the stream actually damaged, out of curiosity, or is the decoder just saying that?

the source (network/disk) may not able to be read fast enough
or the stream has an unknown (to ffmpeg) header (like PaVE (Parrot Video Encapsulation))

Also, would you happen to know about the second thing I ask about? That is the one that I am really trying to figure out at the moment. Do you know what is going on when sws-scale tells you that you have "bad dst image pointers"?

sorry, i am just a ffmpeg user

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 658 days ago

So I am wondering if the PaVE header is placed in front of every frame or every packet that is sent. I assume that they mean it is before every frame. Also, I assume that when the developers guide says "frame" they are talking about one frame of video (ie., one image). With that being said, the PaVE header contains a field for payload_size and I notice that it tends to hover around 18k-24k bytes. Does that seem plausible for an encoded frame of 640 by 368 with color information?

Anyway, I noticed that when I call avcodec_decode_video2, the AVFrame that it is saving the decoded results into has a VERY negative value in the linesize field! Shouldn't it be something more like 640?

When I try to specify a width and height in my AVCodecContext structure, the call to avcodec_decode_video2 results in a segmentation fault.

Has anybody else succeeded in getting all this to work with the drone 2.0? Can you receive the packets, reassemble them into frames, and successfully decode it without issues?

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 658 days ago

Hi, Jeremy, i am not sure about your question, but have you read that video_stage_tcp c file, which is the place where they reassemble the data according to the developer guide. i am wondering how do they relate the data from tcp( which is at vp_com_socket c file)to the vp_api_io_data_t *in ( which seems like input data for reassemble), so far, i can only receive data from port 5555, still stuck there, anyone has ideas about that?

thanks

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 657 days ago

Hey, Jiahui. As far as I can tell, the in pointer there does nothing at the point that you are reading data from the tcp socket. It is just there because that is the general structure that they've set up for their pipeline. Personally, I've been trying to piece together code from the video_com_stage, video_stage_tcp and video_stage_ffmpeg_decoder files to get this done. No success yet, but that is the best that I have been able to make sense of what is going on.

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 656 days ago

Thank you, Jeremy. i am also reading through those few files. If i find out something, i will let you know.

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 656 days ago

Hi, Jeremy, when I look deeply in the data i received,i have the following results(when the data has PaVE)

PaVE
version 2
video codec 4
header size 64
payload size 4552
frame type 3
slice index 0
totalSlice 1
total 9082, header 64, payload 4552
total chunks 1
width 640
height 368
frameNumber 126753
chunk index 0
control 0
streamBytePositionLW 543635959
streamBytePositionUW 0
header1SizeSPS 0
header2SizePPS 0
PaVE
version 2
video codec 4
header size 64
payload size 17602
frame type 1=============================================
slice index 0
totalSlice 1
total 23059, header 64, payload 17602
total chunks 1
width 640
height 368
frameNumber 126755
chunk index 0
control 0
streamBytePositionLW 543644913
streamBytePositionUW 0
header1SizeSPS 14
header2SizePPS 10

Basically, there are two types of frame, 1 & 3, do you know what do they stand for?
The total size is not always equal to headerSize + payloadSize,if it's equal means the frame is complete, if it's not equal, what should i do to these data, or how to make it complete?
And, if it's frame_type 1, the header1Size of sps and header2Size of pps have value : 14,10, why is it so?

if the data doesn't have PaVE, it will be something like this

None
version 229
video codec 45
header size 42379
payload size 2578831804
frame type 192
slice index 74
totalSlice 204
total 25357, header 42379, payload -1716135492
total chunks 67
width 10646
height 40359
frameNumber 1198427378
chunk index 35
control 82
streamBytePositionLW 2501560598
streamBytePositionUW 3699756304
header1SizeSPS 44
header2SizePPS 149

for these kind of data, we just dump them, or they are actually useful?

anyone has idea?

thank you

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 656 days ago

Not sure if I can answer all of your questions, but I'll try to offer some info on ones I think I know how to answer.
Frame type 1 are the I-frames and frame type 3 are the P-frames.
As far as your second question goes, if your buffer has more bytes of data than the header size + the payload size, then you should have a complete frame that you can decode; if your buffer has less than this sum, you need to receive more packets to get the missing data.
I can't answer your third question, unfortunately.
On that last question, I think if you don't have a PaVE header, then you don't have a frame. If a packet has no PaVE header then it is a part of some frame that is currently being sent. I think that means that the header data you posted there at the end is meaningless. Of course, you always know when you have a PaVE header because the first four bytes that make up the header always begin with "PaVE".

Hope any of that helps.

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 653 days ago

thank you. Jeremy. that's useful.

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 653 days ago

Jeremy, if it is a complete frame(1: Has PaVE, 2:total size>=header size + payload size), we can just decode it, right?
i check back video data, those who have PaVE, total size looks like always >= header size + payload size, why is it so? i though for tcp, a frame should be sent by not only one packet?

Thank you.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Jeremy Graham 653 days ago

Yes, if you have a complete frame (which can can determine by those two criteria you listed) then, yes, it can be decoded. As far as the second question goes, it looks like a lot of the time a frame does come in on a single packet as far as I can tell. But it could be broken up across multiple packets, yes. Either scenario could occur.

Hope any of that info helps too.

Jeremy

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 652 days ago

okay, thank you, Jeremy.

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 634 days ago

hi, jeremy, not sure have you got your video or not? if not, maybe you can have a look of the following

https://www.dropbox.com/s/4ki1n0r11069uk0/H264%28c%2Bh%29.zip

basically , if you are using c++, you can just copy the h264 files inside the link, and remember to add avcodec.lib, avformat.lib and swscale.lib to your program.

firstly, call H264->add(receiveBuffer, size );
secondly, call h264->getImage();

By this stage, you should can get the video.

if anyone has a better idea on how to reassemble the frame, please share with me .

thank you .

RE: Ar.Drone 2.0 video data without the SDK? - Added by Christian Saner 629 days ago

I can get the videostream with "ffplay http://192.168.1.1:5555" but when I am sending any command like START: "AT*REF=%d,290718208\r" over UDP port 5556 the drone starts but then the video returns a "End of frame" error and the drone stops sending the videostream.

When I rerun ffplay the video is streamed while the drone is flying but when i send a steering command "AT*PCMD=%d,1,%d,%d,%d,%d\r" the drone again stops the video stream...

Any idea how to force the ArDrone 2.0 to keep sending the stream?

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 628 days ago

hi, Christian, i think it's not a good idea by using ffplay, i tried that before, once my programming was connected to the drone, ffplay displayed nothing.

i think you can take a look of the link which i posted , you need to include those 3 libraries. So far, it works well.

if you have a better idea on how to reassemble the frame, please let me know.
thank you .

RE: Ar.Drone 2.0 video data without the SDK? - Added by Su Demon 619 days ago

The above content is too confusing for us new bird.
Is there anyone can release a c/c++/c# project(for windows) which can receive and decode the video stream sent by ARDrone2.0?
I think we can easily uderstand how the video stream works with the running sample.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 618 days ago

I've been working on getting the video stream under osx and windows and having some progress.

ffplay works for me on osx and linux (using linux under VirtualBox as a test environment)
I can't seem to get GStreamer to load the stream yet, but I think it's just a case of getting the right pipeline.
I've asked https://vimeo.com/46935762 for what GStreamer pipeline they used.

Then it should be fairly straightforwards to get it streaming in both osx and windows into a sample app

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 617 days ago

So now I understand, PaVE (Parrot Video Encapsulation) is what isn't being supported by GStreamer, but is what is working on in ffplay.
I'm curious whether ffplay somehow 'sidesteps' the header, or whether it actually has support for PaVE coded into it.

I've streamed http://192.168.1.1:5555 to a file, and can see 'PaVE' as the first 4 bytes of the stream. This file works in ffplay, but not in GStreamer (yet!).

RE: Ar.Drone 2.0 video data without the SDK? - Added by Dardo Kleiner 617 days ago

GStreamer uses a modified ffmpeg 0.7.2. If you replace ffmpeg in gst-ffmpeg with stock 0.7.2 (ignoring the dire warnings) then gstreamer works. Clearly the gstreamer folks modified the ffmpeg H.264 parser such that it is less forgiving to garbage in the stream. But replacing the ffmpeg library in gst-ffmpeg is not a good solution.

Attached is a little gstreamer plugin that will parse out the PaVE headers and pass the remaining H.264 data down the pipeline. Additionally, if one sets the "latency-drop" parameter it will perform Parrot's recommended TCP latency compensation, dropping frames if any I-Frame is present "upstream" in the data buffer.

$ gcc -fPIC -shared -o libgstpave.so -I. `pkg-config --cflags gstreamer-0.10` gstpaveparse.c `pkg-config --libs gstreamer-0.10`

$ env GST_PLUGIN_PATH=. gst-launch tcpclientsrc host=192.168.1.1 port=5555 ! paveparse ! ffdec_h264 ! fakesink

paveparse.zip - GStreamer PaVE parser (4.9 KB)

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 616 days ago

Wow. Thankyou Dardo.
It's really encouraging to get help in this community.
The ARDrone is such a versatile and capable device.
I'm looking forwards to getting involved with the community here.

i'll check this on my windows machine where i've got an ide setup with gstreamer. on a train now

how did you learn more about the AR Drone?
Is there a wiki anywhere? or other knowledge database (forums?)

my background:
blog.kimchiandchips.com

RE: Ar.Drone 2.0 video data without the SDK? - Added by Dardo Kleiner 616 days ago

That link you sent claims to have replaced the onboard code with Debian/gstreamer. That's pretty darn impressive if you ask me - especially if they're actually flying the thing. I'd think they'd have to have reverse engineered the motor controls, since I've not seen any documentation regarding the actual OS running on the drone (other than the released kernel source).

Otherwise, for coding clients you want to download the Parrot-released SDK - this is the best source of information. If don't want to look at that, spend some time with wireshark/tcpdump and the FreeFlight client. The knowledge base is these forums - much lighter on drone 2.0 info than 1.0 but will improve over time...

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 616 days ago

good point on the vimeo link, i just presumed he meant his debian is talking to the drone.
i can't imagine that he's the only person to get a custom OS running on the drone with cameras and flight, and the only evidence to show for it is a video that could be taken with the stock rom.

has there been any initiative towards onboard code?
obviously we can download the firmware packages (presume this is just the debian install on the drone)
these are plf files. can these be unpackaged?
i presume the issues here are surrounding making the drone autonomous, and therefore having issues with selling them

i'm more interested in remote controlled flight with video feedback for absolute position addressing

trying gstreamer out now....

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 616 days ago

having trouble compiling the code into a dynamic plugin on win32
is there a method for announcing symbols from your own application as element factories to gstreamer?

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 616 days ago

note: presuming it's
gst_plugin_register_static

RE: Ar.Drone 2.0 video data without the SDK? - Added by Dardo Kleiner 616 days ago

Yes, plf can be extracted and modified:

http://code.google.com/p/ardrone-tool/

The onboard distro is not Debian, but a stripped down custom build. The bootloader is custom and unreleased. Nicest thing about 2.0 is that it is a standard-ish OMAP board, and yes at the very least its possible to get a custom bootloader (e.g. xload/u-boot) running, as well as a custom kernel. I have a series of notes on this that I haven't had a chance to clean up and post yet since I managed to recover a bricked drone after installing a bad plf.

Sorry, but I can't really help you with gstreamer on Windows...

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 616 days ago

If I have it statically linked, then i presume I just need to run something like:

    if (!gst_plugin_register_static(GST_VERSION_MAJOR, GST_VERSION_MINOR,
        "paveparse", "Parses PaVE (Parrot Video Encapsulation) streams into h.264.",
        plugin_init_pave,
        "1.0", "LGPL", "ARDrone", "ARDrone",
        "https://projects.ardrone.org/boards/1/topics/show/4282")) {
        cout << "ERROR: PaVE element factory plugin has not been correctly loaded" << endl;
        return 1;
    }

after

gst_init (&argc, &argv);

However, i can't seem to access the plugin_init symbol from your gstpaveparse.c yet.
I'd doing something silly i'm sure.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 616 days ago

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 616 days ago

ok. got it as a plugin. compiling to dll (github updated)

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 616 days ago

Working at
https://github.com/elliotwoods/ARDrone-GStreamer-test/

The plugin now loads (it should be built to lib/gstreamer-0.10 folder in your GStreamer install folder)
Current issue: Complains that it won't link.

Here's my current pipeline:

    /* create elements */
    pipeline = gst_pipeline_new ("main_pipeline");
    source = gst_element_factory_make ("filesrc", "source");
    parser = gst_element_factory_make ("paveparse", "parser");
    decode = gst_element_factory_make ("ffdec_h264", "decoder");
    view = gst_element_factory_make ("sdlvideosink", "view");

    // build graph
    gst_bin_add_many(GST_BIN(pipeline), source, parser, decode, view, NULL);
    if (!gst_element_link_many(source, parser, decode, view, NULL)) {
        std::cout << "Couldn't link elements" << std::endl;
        return 0;
    }

which reports that it can't link the elements

libgstpave.dll - GStreamer Pave parser from Dardo Kleiner compiled for Win32 (36 KB)

RE: Ar.Drone 2.0 video data without the SDK? - Added by Dardo Kleiner 616 days ago

http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/chapter-helloworld.html

This parser is implemented to create its source pad dynamically at runtime. You have to link after starting the pipeline (see use of g_signal_connect in helloworld example).

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 616 days ago

ok i presume command line tests like:

gst-launch filesrc location=E:\\Media\\ARDrone\\record.h264 ! paveparse ! vimux ! filesink location="here
.avi" 

result : https://gist.github.com/3360198
(no file produced)

and

gst-launch filesrc location=E:\\Media\\ARDrone\\record.h264 ! paveparse ! ffdec_h264 ! autovideosink

result: https://gist.github.com/3360248
(no video shown)

should produce something

RE: Ar.Drone 2.0 video data without the SDK? - Added by Dardo Kleiner 616 days ago

The ffdec_h264 example should. Try "gst-launch -v" with a fakesink. You could post your record.h264 somewhere, too.

For avimux the caps are probably not set sufficiently on the pushed buffers.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Dardo Kleiner 616 days ago

--- a/gstpaveparse.c
+++ b/gstpaveparse.c
@@ -284,6 +284,9 @@ gst_paveparse_sink_chain (GstPad * pad, GstBuffer * buf)
         //video_stage_tcp_dumpPave (&pave->header);

         caps = gst_caps_new_simple ("video/x-h264",
+            "width", G_TYPE_INT, pave->header.display_width,
+            "height", G_TYPE_INT, pave->header.display_height,
+            "framerate", GST_TYPE_FRACTION, 25, 1,
             NULL);
         gst_caps_replace (&pave->caps, caps);
         gst_caps_replace (&caps, NULL);

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 616 days ago

gst-launch -v with fakesink:
https://gist.github.com/3360987

here's my h264 as well

thankyou so much for the help!

record.h264 (609.3 KB)

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 616 days ago

this issue seems to be:

0:00:22.374024000  3652   02467180 ERROR                 ffmpeg .:0:: number of
reference frames exceeds max (probably corrupt input), discarding one

RE: Ar.Drone 2.0 video data without the SDK? - Added by Dardo Kleiner 616 days ago

Your video works for me, see attached first frame. Seems like a gstreamer/ffmpeg on Windows issue....

0.jpg - Video frame (24.6 KB)

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 615 days ago

wow
that's bad / good / interesting
i'm going to have a look at the direct ffmpeg solution a little since ffplay seems happy/hacky enough for some situations (but very laggy, i presume if you ignore the framerate and force frames through it might catch up to most recent)

gst-inspect ffdec_h264 reveals version number '0.10.8'
i presume that's the version of the factory rather than the underlying ffmpeg version
but does your version report higher?

Again, thanks for all the invaluable help with this.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 615 days ago

so now i'm working on pulling the required components out of video_stage_ffmpeg_decoder.c (which as you know, handles the PaVE headers)
i feel very close now. i've got ffmpeg compiling on all platforms also so it's a good solution right now.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Elliot Woods 614 days ago

is there a standard way to parse a stream going into ffmpeg?

RE: Ar.Drone 2.0 video data without the SDK? - Added by Igor Fulvi 611 days ago

jiahui chen wrote:

Also no Navigation data feedback? for me, i am also writing my own programming without library. but, i can only control ar drone 2, no nav-data and video feedback. are u using udp? do u have any idea to improve the udp?

I'm trying to write an app without SSK too, but I've no able to reading back any data from Drone, can you help me?
- on which UDDP port I must send the init command?
- some data are sended back from the Drone after the init? on wich port?
- which data I must send to control the Drone?

Thanks in advance!

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 610 days ago

Port 5556, commands sending, udp.
port 5555, video request sending and video data receiving. tcp.
port 5554, navdata request sending and navdata receiving. udp.

these are the 3 most important ports for drone 2, have you download the developer guide? i think you can take a look of that first, inside got command format.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Igor Fulvi 610 days ago

Sì, ho letto la guida sviluppatore SDK, e ho appena scoperto il utilizzo del 5556, 5554 e 5555, così ho provato a ricevere NavData su 5554 inviando i comandi specificati nella SDK, ma di sicuro fatto qualche equivoco, perché non leggo indietro alcun messaggio dalla AR.Drone.
I've found some precisation:
- the UDP port from whom the command would be sent is the same 5554
- after every string sended i must add a the <CR> char at the end (the SDK indicate the LF to be added, but sniffing the packets I've found that the CR is the right char)

So... can you send some info to help me...
I'm sorry for wasting your time!

RE: Ar.Drone 2.0 video data without the SDK? - Added by Igor Fulvi 610 days ago

Yes, I read the SDK developer guide, and I just discovered the use of the 5556, 5554 and 5555, so I tried to get NavData of 5554 by sending the commands specified in the SDK, but it sure made ​​some mistake, because I do not read back any message the AR.Drone.
I've found some precisation:
- the UDP port from whom the command would be sent is the same 5554
- after every string sended i must add a the <CR> char at the end (the SDK indicate the LF to be added, but sniffing the packets I've found that the CR is the right char)

So... can you send some info to help me...
I'm sorry for wasting your time!

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 609 days ago

Igor Fulvi wrote:

Yes, I read the SDK developer guide, and I just discovered the use of the 5556, 5554 and 5555, so I tried to get NavData of 5554 by sending the commands specified in the SDK, but it sure made ​​some mistake, because I do not read back any message the AR.Drone. I've found some precisation: - the UDP port from whom the command would be sent is the same 5554 - after every string sended i must add a the <CR> char at the end (the SDK indicate the LF to be added, but sniffing the packets I've found that the CR is the right char)

So... can you send some info to help me... I'm sorry for wasting your time!

------------------------------------------------------------------------------------------------------------------------------------------------------
hi, are you working with the sdk or writing your own programming for drone 2?
if you are writing your own program, have you created a udp? does this udp work?

RE: Ar.Drone 2.0 video data without the SDK? - Added by Igor Fulvi 609 days ago

I'm writing my program without the use of SDK.
The UDP work, I've been tested between 2 PCs (host 192.168.1.1 and client 192.168.1.2).
I send data to host port 5554 from my port 5554 and the I must disconnect the UDP socket to reconnect it in reading UDP mode listening to port 5554.
I've no data back from drone.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Su Demon 608 days ago

Hi Dardo,I wrote my project in C# with the ffmpeg. Here is my code:
CAVEditLib.CAVConverter converter = new CAVEditLib.CAVConverter();
converter.OutputOptions.VideoBitrate = 2000000;
converter.OutputOptions.VideoCodec = "libx264"; ////{Do not Localize}
converter.OutputOptions.VideoPreset = "libx264-default";
converter.AddTask("http://192.168.1.1:5555",
"E:\\video\\output.mp4");
converter.StartAndWait();
I don't know if I can change the parameters of AddTask from AIFileName into a IP port. And after running this, I can not open the 'output.mp4'.
Can you give me some guide of how to use the ffmpeg to decode the video stream from TCP port.
Thank you very much.

tom sparks wrote:

Dardo Kleiner wrote:

I was successful in decoding frames as they come, using "ffplay http://192.168.1.1:5555". Then I was successful in decoding the live stream within a gstreamer-based (http://gstreamer.freedesktop.org/) application once I replaced its internal ffmpeg with a stock release (0.7.2 in this case was closest to gstreamer's own). A stream is a stream is a stream, if you're sufficiently feeding the "black box" that is the decoder it should just give you the frames when they're ready. In our sniffs I've seen nothing that indicates additional video information being conveyed outside of the TCP/5555 connection.

HTH!

- Dardo

ffplay works :) until the SDK for ar.drone 2.0 comes out this well be my only video option :(

- tom

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 608 days ago

Igor Fulvi wrote:

I'm writing my program without the use of SDK. The UDP work, I've been tested between 2 PCs (host 192.168.1.1 and client 192.168.1.2). I send data to host port 5554 from my port 5554 and the I must disconnect the UDP socket to reconnect it in reading UDP mode listening to port 5554. I've no data back from drone.

------------------------------------------------------------------------------------------------------------------------------------------------------
I think you should try another way, create one UDP, and create two function for the UDP socket, send and receive.
And pass the IP address(192.168.1.1) to the UDP.
Then use the send function to send navdata request.
Use the receive function to receive data.
i think you try to do this first, then see how.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Igor Fulvi 608 days ago

jiahui chen wrote:

Igor Fulvi wrote:

I'm writing my program without the use of SDK. The UDP work, I've been tested between 2 PCs (host 192.168.1.1 and client 192.168.1.2). I send data to host port 5554 from my port 5554 and the I must disconnect the UDP socket to reconnect it in reading UDP mode listening to port 5554. I've no data back from drone.


I think you should try another way, create one UDP, and create two function for the UDP socket, send and receive. And pass the IP address(192.168.1.1) to the UDP. Then use the send function to send navdata request. Use the receive function to receive data. i think you try to do this first, then see how.

In effect I'm just thinking that you suggest! in every way, what are the exact sequence of init string to send to drone (for init)?

RE: Ar.Drone 2.0 video data without the SDK? - Added by Igor Fulvi 607 days ago

Okay, I'm succesed to receive continuos initial navadata.
I must send a packet composed by 4 bytes: 01 00 00 00 to start Drone sending navdata (I've sniffed the comm fron my Android to Drone)
Now: the SDK report this table to identify the navdata...

|Header        Drone state    Sequence number   Vision flag    |Option 1 --------------------------------|Checksum block -------------------------|
|0x55667788                                                    |id              size data       . . .    |cks           id size        cks data   |
|32-bit int.   32-bit int.    32-bit int.       32-bit int.    |16-bit int.     16-bit int.     . . .    |16-bit int.   16-bit int.    32-bit int.|

Now... what's how to decode the "Drone state" and "Vision flag"?
What are the possibile value and meaning of "Option 1"\"id"?
What value I can found in "Option 1" data?

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 606 days ago

hi, Igor,
for the navdata, they have their own structure which is the table you found.

Information inside the Option 1 data (refer to the navdata structure table) are:
uint32_t ctrl_state;
uint32_t vbat_flying;
float32_t theta;
float32_t phi;
float32_t psi;
int32_t altitude;
float32_t vx;
float32_t vy;
float32_t vz;
unit32_ num_frames;

i am not sure whether still hav other information inside the option 1 data, that's all i know.

example Ask for theta (pitch angle)
theta = (float *)(data + 28 );
data = navdata which has been received;
8 bits = one byte;
28 = Header(4 bytes) + Drone state(4 bytes) + Sequence number(4 bytes) + Vision flag(4
bytes) + id(2 bytes) + size(2 bytes) + ctrl_state(4 bytes) + vbat_flying(4 bytes);

RE: Ar.Drone 2.0 video data without the SDK? - Added by Igor Fulvi 606 days ago

Perfetct! Thanks, for now it's a lot of infos.
One more info: do you know how to decode the "Drone state" e "ctrl_state"?

RE: Ar.Drone 2.0 video data without the SDK? - Added by Igor Fulvi 606 days ago

One more trouble: as described in the sdk, after sending one packet to (01 00 00 00) Drone, this enter in bootstrap mode and respond with continuos restricted navdata packets (only status and chksum, no Option1 are send)... and until this point all it's okay for me.
For exiting from bootstrap mode I must send to drone the command
AT*CONFIG=\"general:navdata_demo\",\"TRUE\"\\r

that I've understood as
AT*CONFIG="general:navdata_demo","TRUE"<CR>

at this point (damn!) the drone stop to send any packet!
Reading the SDK seems that I must to send to drone also the sequence number with a command like
AT*CONFIG=123,"general:navdata_demo","TRUE"<CR>

Now... the sequence number that I must send to Drone must be sequence received + 1?
The sequence number that I receive from Drone seems to me strange... I receive some bytes like 74 4A 00 00 (19060 dec), I guess to receive lower sequence number due to the fact that the Drone transmissiones is just started!

RE: Ar.Drone 2.0 video data without the SDK? - Added by jiahui chen 604 days ago

Igor Fulvi wrote:

One more trouble: as described in the sdk, after sending one packet to (01 00 00 00) Drone, this enter in bootstrap mode and respond with continuos restricted navdata packets (only status and chksum, no Option1 are send)... and until this point all it's okay for me. For exiting from bootstrap mode I must send to drone the command [...] that I've understood as [...] at this point (damn!) the drone stop to send any packet! Reading the SDK seems that I must to send to drone also the sequence number with a command like [...] Now... the sequence number that I must send to Drone must be sequence received + 1? The sequence number that I receive from Drone seems to me strange... I receive some bytes like 74 4A 00 00 (19060 dec), I guess to receive lower sequence number due to the fact that the Drone transmissiones is just started!

----------------------------------------------------------------------------------------------------------------------
1. c[ 0 ] = 0x01;
c[ 1 ] = 0x00;
c[ 2 ] = 0x00;
c[ 3 ] = 0x00;
udp->send( c, 4 );

to send navdata request

2. try: sendCommand( "AT*CONFIG=%d,\"general:navdata_demo\",\"FLASE\"\r" ); %d is sequence number.

3. udp->receive.

i think you try again use the above method.

RE: Ar.Drone 2.0 video data without the SDK? - Added by Petr Kout 551 days ago

With some help from other folks, I got the OpenCV work on ARDrone 2.0 on Linux. Here is how I did it:

//petrkout.com/linux/parrot-ardrone-2-0-video-streaming-through-opencv-in-linux/">http://petrkout.com/linux/parrot-ardrone-2-0-video-streaming-through-opencv-in-linux/

Cheers

RE: Ar.Drone 2.0 video data without the SDK? - Added by Petr Kout 551 days ago

Not sure why the create link button didn't work out. Here it is again:

http://petrkout.com/linux/parrot-ardrone-2-0-video-streaming-through-opencv-in-linux

RE: Ar.Drone 2.0 video data without the SDK? - Added by polo polo 49 days ago

How many times himself,<a href="http://www.coach-outletusa.com/"><strong>Coach Outlet Online</strong></a>, let go of it,<a href="http://www.canadagooser.com/"><strong>Canada Goose Jackets</strong></a>, let go of it,<a href="http://www.coachxpursesonsale.com/"><strong>Coach Purses On Sale</strong></a>, why deep powerless among tightly tied,<a href="http://www.guccishoesfactories.net/"><strong>Gucci Outlet Online</strong></a>, tied sinking weak body,<a href="http://www.michaelkors.so/"><strong>Michael Kors Outlet Online</strong></a>, you know,<a href="http://www.polo-outletstore.com/"><strong>Polo Outlet Store Online</strong></a>, some things can not help himself,<a href="http://www.north-faceoutletonlines.net/"><strong>North Face Outlet Online</strong></a>, always happen happen,<a href="http://www.cheaphollistersale.co.uk/"><strong>Hollister Abercrombie Outlet</strong></a>, go,<a href="http://www.coach-factoryy.com/"><strong>Coach Factory Outlet</strong></a>, no matter how sad,<a href="http://www.saclongchamppairs.com/"><strong>Longchamp Sacs Sortie</strong></a>, or failed to keep, even dig up the heart,<a href="http://www.discountbeatsbydresale.com/"><strong>Monster Beats Outlet</strong></a>, pour the love,<a href="http://www.burbagssale2013.com/"><strong>Burberry Bags Outlet</strong></a>, exhausted and blood, but also to no avail,<a href="http://www.ralphwlaurenpolo.com/"><strong>Ralph Lauren UK</strong></a>, find a thousand reasons,<a href="http://www.guccishoesfactories.net/"><strong>http://www.guccishoesfactories.net/</strong></a>, to convince yourself fade,<a href="http://www.ralphlaurenuker.com/"><strong>Polo Ralph Lauren</strong></a>, fade once had, to a fresh start,<a href="http://www.michaelkorstates.com/"><strong>Michael Kors Outlet</strong></a>, but it still can be a stubborn,<a href="http://www.marcjacobsonsale.com/"><strong>Marc Jacobs Bags Outlet</strong></a>, stubborn that unforgettable smiles,<a href="http://www.northsclearance.com/"><strong>North Face Outlet Online</strong></a>, and it was pulling around spin rotation.