You'll need to check the
list of the countries that have the visa exemption
agreement with Japan. If your country is included in the
list, then you will not be required for a visa when entering
Japan under normal circumstances. If your country is not
included in the list, you'll need to obtain a short-term
stay visa from the embassy of Japan in your country.
Please read the Japanese visa guide provided by the ministry of foreign affairs carefully. And be informed that it is the embassy of Japan in your country that makes a decision on issuing a visa. Whether the attendees need Letter of Invitation fully depends on the relations between countries. The conference organizers hereby reiterate that we are neither able to nor authorized to decide the relations between countries.
For more details, please visit: http://www.mofa.go.jp/j_info/visit/visa/index.html.