"In common life, to retract an error, even in the beginning, is no easy task."
So who was the Founding Father who actually did have an affair? This affair lasted over three years and during this time this individual was paying money (blackmailed) to the spouse of the woman he was having an affair with. This founding father was Chief of Staff to George Washington, Secretary of the Treasury, and author of many of the Federalist papers; which was used as an important source of constitutional interpretation. He was killed in a duel — partly because he refused to participate in the shooting. He also made many other great sacrifices for this country. Who was it? Alexander Hamilton.
"In 1791, 23-year-old Maria Reynolds approached the married 36-year-old Alexander Hamilton in Philadelphia, requesting his help. Claiming that her husband James Reynolds had abandoned her and her daughter, Maria asked him for enough money to transport them back to New York City, where her family lived. Hamilton consented, and delivered the money in person to Maria later that night. As Hamilton himself later confessed, "'I took the bill out of my pocket and gave it to her — Some conversation ensued from which it was quickly apparent that other than pecuniary consolation would be acceptable.' The two began an illicit affair that would last at least three years." 
Hamilton is probably most widely known by his authoring of many of the Federalist papers which were mostly written between October of 1787 and August 1788 — which was several years before the affair. When the affair became public, he printed a 95 page pamphlet in which he laid out the sordid details and apologized for his behaviour. Speaking of mistakes, he made the following comment, "In common life to retract an error, even in the beginning, is no easy task."
Hamilton, like all of us — was prone to making mistakes. He did many great things for our country. However, once he engaged in the affair, it is apparent that a different Hamilton emerged. Some years after the affair, John Adams made the following comment about Hamilton after John Adams lost the presidential race, which he partly blamed Hamilton for. Speaking of Hamilton's behaviour, Adams said, "Hamilton, he continued, was the world's greatest schemer and plotter, a man devoid of any moral principal..." 
"After Hamilton became secretary of the Treasury, he began to argue that the welfare clause was a general grant of power, and that Congress could spend tax money or even borrow money for a good cause even though it was not included among the enumerated powers."  In 1936, in the Butler case, the Supreme Court used Hamilton's arguments to justify what had never been allowed before. As a result, many social welfare programs emerged. The United States federal budget went from $6 billion in 1936, to $600 Billion in 1980, and in 2013, to around $3.4 Trillion — approaching a trillion more than it takes in. Recently, in just 7 years, our national debt has exploded from $8.7 Trillion in 2007, to over 17 Trillion in 2014. 
President Grover Cleveland said, "no greater danger confronts us as a nation than the unhappy decadence among our people of genuine love and affection for our Government as the embodiment of the highest and best aspirations of humanity, and NOT as the giver of gifts.”
Democratic President Grover Cleveland
How was the general welfare clause treated prior to 1936? River and harbor bills were vetoed by Presidents Tyler, Polk, Pierce, Grant, Arthur, and Cleveland. Democratic President Grover Cleveland taught, "Individual economy and careful expenditure are sterling virtues which lead to thrift and comfort...
To the extent that the appropriations contained in this bill are instigated by private interests and promote local or individual projects their allowance can not fail to stimulate a vicious paternalism and encourage a sentiment among our people, already too prevalent, that their attachment to our Government may properly rest upon the hope and expectation of direct and especial favors and that the extent to which they are realized may furnish an estimate of the value of governmental care.
I believe no greater danger confronts us as a nation than the unhappy decadence among our people of genuine and trustworthy love and affection for our Government as the embodiment of the highest and best aspirations of humanity, and not as the giver of gifts, and because its mission is the enforcement of exact justice and equality, and not the allowance of unfair favoritism."
In 1921 President Harding taught, “Just government is merely the guarantee to the people of the right and opportunity to support themselves. 
“Just government is merely the guarantee to the people of the right and opportunity to support themselves.”
President Warren Harding
When Congress wanted to give a sum of money to a widow, Congressmen Davy Crockett (1827-1831 and 1833-1835) went to the floor of congress and said the following, "Mr. Speaker — I have as much... sympathy... as any man in this House, but... Congress has no power to appropriate this money as an act of charity. Every member upon this floor knows it. We have the right, as individuals, to give away as much of our own money as we please in charity; but as members of Congress we have no right so to appropriate a dollar of the public money."
I would encourage you to take the time to learn more about Alexander Hamilton. His wit and wisdom was definitely unique. Take time to read the Federalist Papers. You may find many of the arguments made then are relevant now.
: http://en.wikipedia.org/wiki/Hamilton %E2%80%93Reynolds_sex_scandal
: Citizen Hamilton, The Wit & Wisdom of an American Founder, Donald R. Hickey and Connie D. Clark, p. 99, ISBN 0-7425-4975-5
: John Adams, A Life, John Ferling, p. 394, ISBN: 978-0-19-539866-3
: The Making of America - The Substance and Meaning of the Constitution, W. Cleon Skousen, p. 388 ISBN: 0-88080-017-8
: Video-I.O.U.S.A.: Byte-Sized - The 30 Minute Version, http://www.youtube.com/watch?v=O_TjBNjc9Bo
: Grover Cleveland, Letter to the House of Representatives, May 29, 1896
: The Making of America - The Substance and Meaning of the Constitution, W. Cleon Skousen, p. 391 ISBN: 0-88080-017-8
: The Making of America - The Substance and Meaning of the Constitution, W. Cleon Skousen, p. 391 ISBN: 0-88080-017-8, National Center for Constitutional Studies - http://www.nccs.net