| ||
Pass formatting specifiers into the application and see if hexadecimal values are returned. For example, if you have an application that expects a file name and returns an error message containing the input when the file cannot be found, then try giving it file names like NotLikely%x%x.txt . If you get an error message along the lines of "NotLikely12fd234104587.txt cannot be found," then you have just found a format string vulnerability.
This is obviously somewhat language-dependent; you should pass in the formatting specifiers that are used by the implementation language youre using at least. However, since many language run times are implemented in C/C++, youd be wise to also send in C/C++ formatting string commands to detect cases where your underlying library has a dangerous vulnerability.
Note that if the application is web based and echoes your user input back to you, another concern would be cross-site scripting attacks.