Bring the stroller and a snack for the kids.
Knoxville Museum of Art is closed on Mondays, so avoid going then. On weekends, arrive early in the morning, around 9:30 a.m. or 10 a.m. to start the day's activities. Saturdays will naturally be busy, but Knoxville is a small town, so it's not bad unless there is a huge event like “Rhythm and Blues” (which takes place the first weekend of April).
March through May are terrific times to visit Knoxville, with festivals and events happening nearly every weekend and temperatures being pleasantly warm but not hot.