Too Big?
The upper mass limit for a main sequence star is somewhere around 100 - 200 times that
of the sun. This is thought to be due to the enormous energy contained within a collapsed
cloud of this size, that it would simply be unstable to its own radiation (luminosity).
Stars have been confirmed to exist up to masses of 60 times the Sun, and other indicators suggest stars as high as 100x the Sun exist. It is important to keep in mind that massive stars are more rare to form AND live the shortest life. There is a great bias against finding what might be the most massive star possible.
Too Small?
This is a bit easier to quantify. We define a star to be a self illuminating object which derives its energy from nuclear fusion reactions. If a star is too small (low in mass), its central temperature and density will never achieve sufficiently high values to begin nuclear fusion reactions. These objects are called, Brown Dwarfs.
Brown Dwarfs where first conceived of in the early 1960s. Stellar models suggest a true star must have a mass of at least 80 times Jupiter to fuse hydrogen. These objects where suspected to exist, but would be very difficult to find due to their very low luminosity. However, today, a number of Brown Dwarf objects have been found with very large modern telescopes.
What's less massive than a Brown Dwarf?
A planet. Here the convention is less quantitative, but a few times the mass of Jupiter is
a good upper mass limit for a planet. The lower limit for a planet is Pluto's mass.